ESGF P2P Node Projects
The ESGF P2P Node is the central project. It is an amalgam of several
sub-projects and external software components and libraries to create a single
Data Node artifact capable of searching, sharing and posting large amounts of
metadata.
installer
- Build, install, configure and maintain the ESGF Node.
node-manager
- Central component for managing constituent components and overall inter
and intra node activity.
- The node manager provides the framework for the coordination and management of the various components that comprise the ESGF Node.
- coordination, monitoring, metrics, notification, routing, discovery, etc
esgf-shell
- The shell provides an interactive commandline (REPL) environment from which
the user can manipulate the node and take advantage of node features.
- Use the esgf shell (esgf-sh) as a command line alternative interface into all
the features of the ESGF Node. Write your own custom commands in Java, Clojure,
Scala, Groovy and/or BeanShell. The shell provides familiar shell features and
full set of commands to let you take advantage of the simple power of the
command line.
- The esgf shell provides a simple shell and command line interface to the
node’s services. Users can use the shell to perform any of the tasks that may
be performed through the web user interface. To help end users leverage the
shell to it’s maximum the shell provides the ability for end-users to develop
custom shell commands. Commands can be written in Java, Closure, Scala,
Jython, Groovy, or BeanShell. Through custom commands end-users have a direct
mechanism for extending the node features and creating new ones!
- Power, simplicity, flexibility, extensibility, etc.
dashboard
- Federation level monitoring tool. The dashboard and the node manager enjoy a
symbiotic relationship as they both participate in federation level
activities. Different from the node manager’s role the dashboard monitors
nodes once they are discovered and join the federation. The dashboard
maintains historical information, whereas the node manager is primarily
interested in keeping federation data up to date and in sync. The dashboard
has a ‘back-end’ written and a quite attractive :-) front-end displaying not
only node participation but node location as well!
desktop
- Federation level desktop
- The ESGF Desktop extends the functionalities provided by the data node
adding an intuitive interface to back-end esgf node “monitoring” statistics
and information - using the desktop metaphor. It extends the data node by
providing a back-end monitoring system and a front-end user interface
for monitoring purposes.
publisher
- Harvests, publishes, replicates and maintains meta data.
- Primary feature component for performing meta data harvesting, publishing
and maintenance. Features pluggable handler design architecture for multi
data format support.
idp
- The identity provider in the ESGF SSO federation as well as account management
and access control system. Also including a web-based interface for site
administrators.
- Identity Provider and User Manager
- Where users are managed and identities kept.
- identity, security, naming, access, people
- Identity Provider and User Manager
- Where users are managed and identities kept.
- identity, security, naming, access, people
orp
- The OpenID Relying Party - relays end-users to their home organization’s IDP
(identity provider) for secure login and connection to local resources
- OpenID relying party
- Provides the federation of nodes the ability to have single sign-on, with
OpenID support.
- Single sign-on
security
- Security infrastructure for nodes and federation.
- Security infrastructure protecting data integrity and access on each node and
across the federation.
search
- Search mechanism that supports search across entire corpus of ever-changing
published metadata.
- The ESGF search module contains the next generation search functionality for
the Earth System Grid Federation, built upon the Apache Solr search engine. The
package contains functionality for:
- Publishing and unpublishing search metadata records into and from a Solr
server. Metadata records are generated by harvesting a remote metadata source
(a hierarchy of THREDDS catalogs, a OAI repository, a CAS metadata catalog).
- Searching the Solr engine content via a free text, faceted search or a
geo-spatial and temporal search.
drslib
- Set of libraries for handling the DRS taxonomy that model data adheres to.
stager
- A system for connecting to and retrieving data from back-end deep storage
archives.
- The Stager provides the means to transparently access files located at a
remote system. Remote files are cached locally preserving the directory
structure from the remote structure.
- Implementation
- The current implementation uses a Java web Filter for intercepting the file
requests targeted at servlets running locally. These calls are handled
differently depending on the user-agent attribute from the http request
header.
- If the request was started from within a browser, the user is redirected
to a “waiting page” from where it will be automatically redirected to the
original servlet when the file is ready.
- In case the request was started by a tool, the request will just block
until the file is available.
- The file retrieval will be triggered only once and in cache the file is
already in the cache the filter will return immediately allowing the call to
proceed normally without any delay. This call is logged to optimize the
cache-cleaning mechanism.
web-fe
- The web front-end to the federation. Interface to fasceted search and other services.
fetcher
- Download published datasets and stay up-to-date.
- Fetching and keeping updated with ESGF Published data.
- Fast, Secure and Easy
- Stay Tuned… - Work in progress
- The ESGF Fetcher leverages the ESGF security mechanisms (esgf-security) and
provides fast and secure HTTP file downloads. It also maintains and updates
downloaded files with its built-in “broadcatching” capabilities. The fetcher has
a secured RESTful interface as its control channel and provides a web and esgf
shell (esgf-sh) command line interface. The Fetcher is a stand alone service
that may be run on any host without any other dependencies.
- Algorithms and tools to help you wrangle your data.
- The Premiere Toolbox of computation algorithms
- Big Data -> Big Computation -> Big Analytics… No Problem
- Stay Tuned… - Work in progress
- The ESGF Toolbox is a collection of algorithms written in Java, Scala, Clojure
and Python that leverages ESGF Compute Tools’ “Big Data” infrastructure to
perform computation, analytics and querying over ESGF datasets.
