A framework for creating and deploying Apache Storm streaming computations with less friction.
Definition
We don't have any change log information yet for version 4.118 of f.lux. Sometimes publishers take a little while to make this information available, so please check back in a few days to see if it has been updated. Therefore, 1.1.2 means rosin flux with halides. The older specification, still often in use for specifying fluxes in shops, is the older German DIN 8511; the pairing is, however, not always one-to-one (note the multiple vs. One relation old standard vs. New standard in table below).
flux |fləks| noun
- The action or process of flowing or flowing out
- Continuous change
- In physics, the rate of flow of a fluid, radiant energy, or particles across a given area
- A substance mixed with a solid to lower its melting point
Rationale
Bad things happen when configuration is hard-coded. No one should have to recompile or repackage an application inorder to change configuration.
About
Flux is a framework and set of utilities that make defining and deploying Apache Storm topologies less painful anddeveoper-intensive. One of the pain points often mentioned is the fact that the wiring for a Topology graph is often tied up in Java code,and that any changes require recompilation and repackaging of the topology jar file. Flux aims to alleviate thatpain by allowing you to package all your Storm components in a single jar, and use an external text file to definethe layout and configuration of your topologies.
Features
- Easily configure and deploy Storm topologies (Both Storm core and Microbatch API) without embedding configurationin your topology code
- Support for existing topology code (see below)
- Define Storm Core API (Spouts/Bolts) using a flexible YAML DSL
- YAML DSL support for most Storm components (storm-kafka-client, storm-hdfs, storm-hbase, etc.)
- Convenient support for multi-lang components
- External property substitution/filtering for easily switching between configurations/environments (similar to Maven-style
${variable.name}
substitution)
Usage
To use Flux, add it as a dependency and package all your Storm components in a fat jar, then create a YAML documentto define your topology (see below for YAML configuration options).
Building from Source
The easiest way to use Flux, is to add it as a Maven dependency in you project as described below.
If you would like to build Flux from source and run the unit/integration tests, you will need the following installedon your system: Marked 2 2 5 21.
- Python 2.7.x or later
- Node.js 0.10.x or later
Building with unit tests enabled:
Building with unit tests disabled:
If you would like to build Flux without installing Python or Node.js you can simply skip the unit tests:
Note that if you plan on using Flux to deploy topologies to a remote cluster, you will still need to have Pythoninstalled since it is required by Apache Storm.
Building with integration tests enabled:
Packaging with Maven
To enable Flux for your Storm components, you need to add it as a dependency such that it's included in the Stormtopology jar. This can be accomplished with the Maven shade plugin (preferred) or the Maven assembly plugin (notrecommended).
Flux Maven Dependency
The current version of Flux is available in Maven Central at the following coordinates:
xml<dependency> <groupId>org.apache.storm</groupId> <artifactId>flux-core</artifactId> <version>${storm.version}</version></dependency>
Using shell spouts and bolts requires additional Flux Wrappers library:
xml<dependency> <groupId>org.apache.storm</groupId> <artifactId>flux-wrappers</artifactId> <version>${storm.version}</version></dependency>
Creating a Flux-Enabled Topology JAR
The example below illustrates Flux usage with the Maven shade plugin:
Deploying and Running a Flux Topology
Once your topology components are packaged with the Flux dependency, you can run different topologies either locallyor remotely using the
storm jar
command. For example, if your fat jar is named myTopology-0.1.0-SNAPSHOT.jar
youcould run it locally with the command:Command line options
![Flux Flux](https://p1-e6eeae93.imageflux.jp/c!/w=1200,h=630,a=3,u=1/cdkatabuki/418d68c52d2526806f43.png)
NOTE: Flux tries to avoid command line switch collision with the
storm
command, and allows any other command lineswitches to pass through to the storm
command.For example, you can use the
storm
command switch -c
to override a topology configuration property. The followingexample command will run Flux and override the nimbus.seeds
configuration:Sample output
YAML Configuration
Flux topologies are defined in a YAML file that describes a topology. A Flux topologydefinition consists of the following:
- A topology name
- A list of topology 'components' (named Java objects that will be made available in the environment)
- EITHER (A DSL topology definition):
- A list of spouts, each identified by a unique ID
- A list of bolts, each identified by a unique ID
- A list of 'stream' objects representing a flow of tuples between spouts and bolts
- OR (A JVM class that can produce a
org.apache.storm.generated.StormTopology
instance:- A
topologySource
definition.
- A
For example, here is a simple definition of a wordcount topology using the YAML DSL:
Property Substitution/Filtering
It's common for developers to want to easily switch between configurations, for example switching deployment betweena development environment and a production environment. This can be accomplished by using separate YAML configurationfiles, but that approach would lead to unnecessary duplication, especially in situations where the Storm topologydoes not change, but configuration settings such as host names, ports, and parallelism paramters do.
For this case, Flux offers properties filtering to allow you two externalize values to a
.properties
file and havethem substituted before the .yaml
file is parsed.To enable property filtering, use the
--filter
command line option and specify a .properties
file. For example,if you invoked flux like so:With the following
dev.properties
file:You would then be able to reference those properties by key in your
.yaml
file using ${}
syntax:In this case, Flux would replace
${kafka.zookeeper.hosts}
with localhost:2181
before parsing the YAML contents.Environment Variable Substitution/Filtering
Flux also allows environment variable substitution. For example, if an environment variable named
ZK_HOSTS
if defined,you can reference it in a Flux YAML file with the following syntax:Components
Components are essentially named object instances that are made available as configuration options for spouts andbolts. If you are familiar with the Spring framework, components are roughly analagous to Spring beans.
Every component is identified, at a minimum, by a unique identifier (String) and a class name (String). For example,the following will make an instance of the
org.apache.storm.kafka.StringScheme
class available as a reference under the key'stringScheme'
. This assumes the org.apache.storm.kafka.StringScheme
has a default constructor.Static factory methods
It is also possible to use static factory methods from Flux. Given the following Java code:
it is possible to use the factory methods as follows:
Contructor Arguments, References, Properties and Configuration Methods
![1/2 1/2](https://i.ytimg.com/vi/QvkpkCfSGS8/sddefault.jpg#404_is_fine)
Constructor Arguments
Arguments to a class constructor can be configured by adding a
contructorArgs
element to a components.constructorArgs
is a list of objects that will be passed to the class' constructor. The following example creates anobject by calling the constructor that takes a single string as an argument:References
Each component instance is identified by a unique id that allows it to be used/reused by other components. Toreference an existing component, you specify the id of the component with the
ref
tag.In the following example, a component with the id
'stringScheme'
is created, and later referenced, as a an argumentto another component's constructor:N.B.: References can only be used after (below) the object they point to has been declared.
Properties
In addition to calling constructors with different arguments, Flux also allows you to configure components usingJavaBean-like setter methods and fields declared as
public
:In the example above, the
properties
declaration will cause Flux to look for a public method in the SpoutConfig
withthe signature setIgnoreZkOffsets(boolean b)
and attempt to invoke it. If a setter method is not found, Flux will thenlook for a public instance variable with the name ignoreZkOffsets
and attempt to set its value.References may also be used as property values.
Configuration Methods
Conceptually, configuration methods are similar to Properties and Constructor Args -- they allow you to invoke anarbitrary method on an object after it is constructed. Configuration methods are useful for working with classes thatdon't expose JavaBean methods or have constructors that can fully configure the object. Common examples include classesthat use the builder pattern for configuration/composition.
The following YAML example creates a bolt and configures it by calling several methods:
The signatures of the corresponding methods are as follows:
Arguments passed to configuration methods work much the same way as constructor arguments, and support references aswell.
Using Java enum
s in Contructor Arguments, References, Properties and Configuration Methods
Flux 6
You can easily use Java
enum
values as arguments in a Flux YAML file, simply by referencing the name of the enum
.For example, Storm's HDFS module includes the following
enum
definition (simplified for brevity):And the
org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy
class has the following constructor:The following Flux
component
definition could be used to call the constructor:The above definition is functionally equivalent to the following Java code:
Topology Config
Flex 6 11 Upgrade
The
config
section is simply a map of Storm topology configuration parameters that will be passed to theorg.apache.storm.StormSubmitter
as an instance of the org.apache.storm.Config
class:If you have existing Storm topologies, you can still use Flux to deploy/run/test them. This feature allows you toleverage Flux Constructor Arguments, References, Properties, and Topology Config declarations for existing topologyclasses.
The easiest way to use an existing topology class is to definea
getTopology()
instance method with one of the following signatures:or:
You could then use the following YAML to configure your topology:
If the class you would like to use as a topology source has a different method name (i.e. not
getTopology
), you canoverride it:N.B.: The specified method must accept a single argument of type
java.util.Map<String, Object>
ororg.apache.storm.Config
, and return a org.apache.storm.generated.StormTopology
object.Spouts and Bolts
Spout and Bolts are configured in their own respective section of the YAML configuration. Spout and Bolt definitionsare extensions to the
component
definition that add a parallelism
parameter that sets the parallelism for acomponent when the topology is deployed.Because spout and bolt definitions extend
component
they support constructor arguments, references, and properties aswell.Shell spout example:
Kafka spout example:
Bolt Examples:
Streams and Stream Groupings
Streams in Flux are represented as a list of connections (Graph edges, data flow, etc.) between the Spouts and Bolts ina topology, with an associated Grouping definition.
A Stream definition has the following properties:
name
: A name for the connection (optional, currently unused)from
: The id
of a Spout or Bolt that is the source (publisher)to
: The id
of a Spout or Bolt that is the destination (subscriber)grouping
: The stream grouping definition for the StreamA Grouping definition has the following properties:
type
: The type of grouping. One of ALL
,CUSTOM
,DIRECT
,SHUFFLE
,LOCAL_OR_SHUFFLE
,FIELDS
,GLOBAL
, or NONE
.streamId
: The Storm stream ID (Optional. If unspecified will use the default stream)args
: For the FIELDS
grouping, a list of field names.customClass
For the CUSTOM
grouping, a definition of custom grouping class instance Drive scope 1 2 800.The
streams
definition example below sets up a topology with the following wiring:Custom Stream Groupings
Custom stream groupings are defined by setting the grouping type to
CUSTOM
and defining a customClass
parameterthat tells Flux how to instantiate the custom class. The customClass
definition extends component
, so it supportsconstructor arguments, references, and properties as well.The example below creates a Stream with an instance of the
org.apache.storm.testing.NGrouping
custom stream groupingclass.Includes and Overrides
Flux allows you to include the contents of other YAML files, and have them treated as though they were defined in thesame file. Includes may be either files, or classpath resources.
Includes are specified as a list of maps:
If the
resource
property is set to true
, the include will be loaded as a classpath resource from the value of thefile
attribute, otherwise it will be treated as a regular file.The
override
property controls how includes affect the values defined in the current file. If override
is set totrue
, values in the included file will replace values in the current file being parsed. If override
is set tofalse
, values in the current file being parsed will take precedence, and the parser will refuse to replace them.N.B.: Includes are not yet recursive. Includes from included files will be ignored.
Basic Word Count Example
This example uses a spout implemented in JavaScript, a bolt implemented in Python, and a bolt implemented in Java
Topology YAML config:
Micro-Batching (Trident) API Support
Currenty, the Flux YAML DSL only supports the Core Storm API, but support for Storm's micro-batching API is planned.
To use Flux with a Trident topology, define a topology getter method and reference it in your YAML config:
Flux is an innovative application for Mac OS X that will simplify the design and layout of web pages thanks to a visual editor for web pages that allows you to insert all kinds of elements just by dragging and dropping.
This can be used to generate static pages or dynamic pages that make use of technologies like PHP, AJAX or Ruby on Rails. The two languages that allows Flux to make a page or XHTML and CSS, so the resulting website can be accessed by practically any browser.
Flux also facilities the creation of styles and their subsequent application to any element on a page. The sidebar on the left lists all the styles in the style sheet relating to the project. Just drag the label on an item to apply the appropriate style you need.
Begin a project from scratch or part of the templates that Flux has available to familiarize yourself with the program.
NotesThis can be used to generate static pages or dynamic pages that make use of technologies like PHP, AJAX or Ruby on Rails. The two languages that allows Flux to make a page or XHTML and CSS, so the resulting website can be accessed by practically any browser.
Flux also facilities the creation of styles and their subsequent application to any element on a page. The sidebar on the left lists all the styles in the style sheet relating to the project. Just drag the label on an item to apply the appropriate style you need.
Begin a project from scratch or part of the templates that Flux has available to familiarize yourself with the program.
Flux 6 1 1 Equals
For Mac OS X 10.5 or later. Users of Mac OS X Tiger (10.4.9) must download Flux 1.4.91