The Marionette Collective, also known as MCollective, is a framework for building server orchestration or parallel job-execution systems. Most users programmatically execute administrative tasks on clusters of servers.
MCollective has some unique strengths for working with large numbers of servers:
- Instead of relying on a static list of hosts to command, it uses metadata-based discovery and filtering. It can use a rich data source like PuppetDB, or can perform real-time discovery across the network.
- Instead of directly connecting to each host (which can be resource-intensive and slow), it uses publish/subscribe middleware to communicate in parallel with many hosts at once.
To get an immediate feel for what this means, check out the videos on the Screencasts page. Then, keep reading below for further information and links.
We’ve also created a Vagrant-based demo, where you can easily experiment with MCollective.
What is MCollective, and What Does It Allow You to Do?
- Interact with clusters of servers, whether in small groups or very large deployments.
- Use a broadcast paradigm to distribute requests. All servers receive all requests at the same time, requests have filters attached, and only servers matching the filter will act on requests. There is no central asset database to go out of sync, because the network is the only true source.
- Break free from identifying devices through complex host-naming conventions, and instead use a rich set of metadata provided by each machine — from Puppet, Facter, or other sources — to address them.
- Use simple command-line tools to call remote agents.
- Write custom reports about your infrastructure.
- Use agent plugins to manage packages, services, and other common components created by the community.
- Write simple RPC style agents, clients, and web UIs in Ruby.
- Extremely pluggable and adaptable to local needs.
- Leverage rich authentication and authorization models in middleware systems as a first line of control.
- Include fine-grained authentication using SSL or RSA, authorization, and request auditing. For more information, see the Security Overview.
- Re-use middleware features for clustering, routing, and network isolation to realize secure and scalable configurations.
MCollective provides a stable core framework that you can build into a system to meet your own needs. MCollective is pluggable in the following areas:
- Replace our STOMP-compliant middleware with your own, such as something AMQP-based.
- Replace our authorization system with one that suits your local needs.
- Replace our Ruby Marshal and YAML-based serialization with your own, such as cross-language JSON Schema.
- Add data sources. It’s easy to use Puppet’s data, and you can configure or build plugins for other data sources.
- Create a central inventory of services leveraging MCollective as transport that can regularly run and distribute inventory data.
MCollective is licensed under the Apache License, Version 2.0.
Next Steps and Further Reading
Introductory and Tutorial Pages
- See it in action in our Screencasts.
- Read the Overview of Components to understand MCollective’s basic structure.
- Use the Vagrant Demo Environment to try MCollective.
- See the Standard Deployment Getting Started Guide to install and deploy MCollective in your own infrastructure.
- Read the Terminology page if you see any words where the meaning in the context of MCollective is not clear.
- Read the change log and release notes to see how MCollective has developed.
- Learn how to write basic node reports for your servers.
- Learn how to write simple agents and clients using our Simple RPC Framework.
- MCollective’s author has several blog posts about MCollective.
- Subscribe and post questions to the mailing list.