Information for contributors to AMT Software projects
History
The 9-year journey
Ace Monster Toys was founded as a hackerspace. And hack we did. The original platform was attached to a drupal site, pin payments, a google instance, apps, a gordian knot of services and a custom php “framework” dumping data into text files on a cloud provider. And it was grown over time by many contributors in all their favorite flavors of code and methodology. It is a testament to the work people did that it didn’t fail more often. There was a pattern of what we have dubed “gold brick” contributions. They built it, it did the job, nothing was documented and when it broke they had moved on to greener pastures. So, it was valuable (gold) but hurts like hell when it breaks hits you (brick).
The last four years
In February of 2016, we moved the membership management site (aka the Main AMT site) to WordPress (WP) and WooCommerce (WC) hosted with WPEngine and moved member discussions to Slack from a mailman listserv. This allowed non-coders to provide operational admin and member support to the org. It had many benefits but that was the main one. We integrated it with the establish backend framework hosted on prgrm.com. That framework was a proto version of AMT Grand Central.
Once the newer more easy to use systems were up, we began developing Slack integrations (toybot) and better communication platform integrations. We learned how to work as a team in the managed WPE environment and for a while got a bit spoiled with functional operations tools like billing and logs.
Alas, all good things must come to an end. The framework that supported access systems and logs started to catastrophically fail. We made the decision to abandon it and build a modern tool (AMTGC). This effort began in 2017 when we got serious about the FATT project.
We have gone through 3 major false starts were months were burned up on tools that turn out not to work for us. We had great contributors but not a lot of actual closing of projects beyond the contractor supported WP/WC customizations. Some of the major factors while large efforts did not produce anything despite folks best intentions:
- Lack of clarity about working methods and norms at the start
- Over-engineering of the MVP
- Adoption of tools that were hard to use and more than we needed (docker)
- Inability to set up new contributors (ie instructions for setting up local dev environments)
- Attrition of contributors (loss of interest)
- Contributors moving or have major life changes
- Personality clashes among developers
Through all the pitfalls many good outcomes including lessons learned were gleaned in this time period as well. As we move forward we seek to leverage those learnings.
The Great Wiki Data Loss of 2019
July 7th, 2019 we had a well-meaning former member accidentally overwrite the data vol. With the root when trying to restore functionality. The disk was full. The result was 18 months of lost documentation much of it operational. We are still feeling the pain.
The loss was caused when standards were not set during a wiki-upgrade project that became a bit over-engineered. The live (aka production) wiki was moved to a member personal server space and AMT folks continued to add and edit content to the wiki. This instance was not managed or backed up automatically. The upgraded version of the wiki never came to fruition due to attrition by contributors — something that is very common with volunteers.
Leaders were aware of the situation but were at the mercy of the now-former member hosting the production version of the wiki. Requests were made over 6 months to move the instance back to AMT managed and backed up space. Disaster struck before that happened. We are all paying the price.
Mission
The mission of the AMT Software group is to create and maintain operational tools for the organization including access systems, member services, and program support.
Guiding Principles
AMT has unique technology needs and support abilities. Selecting optimal technology for AMT is not the same as selecting it for a corporate environment or a personal project. The list below will help developers embrace common understanding of why AMT made certain choices. These principles guide choices and decisions for the group.
All production instances exist on AMT owned infrastructure
We don’t run official AMT Software attached to any operational functionality on personal servers. Ever. It is be hosted on am AMT (e.g. LLC) owned instance and pointed to by an AMT owned URL. If it’s not on an AMT resource, it’s not a production instance.
Only a small number of trusted individuals have production access
Direct shell access to production accounts is typically not needed except in very specific and sensitive cases. The board and officers ultimately control the flow of these credentials. This protects AMT from well-intended accidents.
No docker/containers
We don’t have sufficient need to warrant excess infrastructure. While this may change in the future, it removes skill tree dependencies and lowers barriers to entry as well as debugging.
Communication
Being chatty on slack is a good thing. If you are impacting any shared resource, announcing what you will do, when you will do it, when you start, and when you finish are all welcome and examples of great communication. Timely communication of facts is extremely important.
Information Confidence
Confidence in the factual content of listed information enables development. Hard links to information, documentation, specifications, instructions, plans, models, etc are all great. Reviews of documentation and links are extremely helpful, especially by new contributors!
Source Control
All production source code must be version controlled on AMT owned infrastructure (external or internal).
Continuity
When making any change such as developing a new feature or selecting new infrastructure, consider how it will live at AMT given that AMT resources are in flux. New technology should always be common, supported, easy to google, well documented, have a large support community, etc. When creating new code or devices, they should be fully documented before being considered “finished”.
Current Goals
We have three goals. In priority they are:
- Replace failing operational systems
- Automate as many operational functions as is possible
- Improve operations for the new and future scale of the organization
Replace failing operational systems
Systems fail, technology changes, and things eventually have to be replaced over time. This is the nature of software. To best understand why AMT systems are failing, read the historical overview in this document. When folks join the team or are making design decisions they should first endeavor to understand what the previous systems did for the organization as well as how and why those functions are performed today.
Here are some of the rapidly failing systems in order of catastrophic failure being felt (aka how much pain it is causing):
- Access points (aka doors) – These allow RFID based access to the AMT spaces and laser. This is also known as the FATT Project. Currently, one or more door systems have been down for over 6 months.
- Logging functions – This allows folks helping members to troubleshoot access issues or diagnose issues in member use. This has been down for over 4 months.
- Laser billing – This allows us to bill for laser time. This system has been down partially for over 6 months and totally down for 4 months. Which means we are not collecting the laser fees.
Automate as many operational functions as is possible
AMT is currently experiencing a sustainability crisis. Our growth outpaced our volunteer contributions and our operational programs didn’t scale well. The automation that would have the greatest impact if implemented are:
- Use metrics reporting
- Preventive maintenance notifications
- Contribution tracking
- Bill capture for honor bar
- Bill generation for honor bar and laser
The second tier automation are:
- Financial reporting for programs
- Task tracking and assignment in Monster Corps
- Safety data in spaces
- Social media and web content publication
Automations are implemented based on the greatest impact or ease of implementation based on resources available… ie. do we have a qualified volunteer or can we afford a contract developer.
Improve operations for the new and future scale of the organization
Everyone wants more, better. Smarter machines, better access, and tracking, more services, etc. Some of the dreams currently on the table are:
- Improve use metrics tracking with sensor and data logs, including analysis dashboards
- Implement the vending machine
- Implement the honor bar kiosk
- Support FATT Implementation across all major tools and spaces
Working Style
AMT has always given folks a lot of latitude to be themselves and do things in the way they are most comfortable. But as we have grown we have fostered better and better skills for sharing. These gains are being leveraged to have better working teams for AMT software development yielding better results for the community. The AMT Software contributors started out with some base agreements about how things will be done.
In all things communicating and treating each other with respect per the AMT Social Contract.
Who does what
Decision Making
Different people make different decisions based on their role and subject matter expertise. Most decisions are made using the consultative method. To learn more about AMT decision-making models read this blog post.
Project Management
Identified project managers assign tasks or folks self-select for a task with the project managers’ approval.
Coding
Lost of folks do the coding. Some folks validate code and documentation.
Maintenance
Currently, maintenance is what Crafty can do or solicit other generous folks to do.
Platforms/Languages
Coding
- When choosing a language or platform a minimum of two long term stable contributors need to be familiar if not an expert. The preference is for common accessible languages.
- When setting up environments all environments need to include at minimum a development and production instance that are backed up.
- All code is documented in the AMT Github instance. The repo depends on the project. Repo’s should include “read me” and readme’s should include:
- How to set up for development
- Where the software runs
- Installation instructions
- Where to find any credentials the software needs to run
- Attribution and links to the source material or dependancies
Platforms
- WPEngine – A managed WordPress host. They host acemonstertoys.org, wiki.acemonstertoys.org, and foballthethings.org – We use WooCommerce with extensions.
- Prgmr.com – A virtual server that hosts the legacy framework. Some access systems still depend on it for white lists.
- Digital Ocean – Hosts the current version of AMT Grand Central
- Azure – The future home of AMT Grand Central
Services
- DocuSign – for legal documentation. There is an integration on the AMT Main site.
- [Further documentation as time allows coming soon-ish]
Getting Started
It is currently very hard to get started. We know this. The AMT Sustainability project is looking to make this better bit by bit. As does this document.
Approach
The idea when a contributor first starts is to start with a small manageable project or task from the backlog that matches their existing skills. This allows for:
- Trust to be built within the group for the new contributor
- Exposes the new contributor to the AMT Software development landscape
Ideally, a new contributor would be paired with an existing contributor for peer learning specific to goals and skillsets. Right now we have an availability issue with buddies or mentors.
Learning on the project is encouraged; the reality right now (Oct. 2019) is we need working solutions desperately. We don’t have the time nor financial resources to indulge in longer learning curves in fundamental skill-sets.
Credentials and access to platforms, data, and services
Contributors, especially folks new to AMT are given only the access they need to contribute. Access to production instances of most tools is very very limited. See The Great Wiki Data loss of 2019. Possible levels of access:
- Access to AMT Main site dev or staging
- Azure Administrative access
Etiquette
Primary Website – WP Engine
- Production is well, production – Limited access to production
- Dev is a true development environment – more open access to dev. – requires access accounts and cooperation between developers.
- Staging is a copy of production where features ready to be migrated to production are tested – This requires that staging be kept up to day with production
Basic workflow
- Create asana task for what needs to be developed
- Agree on due date and an understanding of the task (and time if developer is a contractor)
- Develop locally and/or at devamt.wpengine.com
- Get Ace Lead/Requester Approval OR Iterate until it works. The asana task is tagged “Approve for Production”
- Once Approved by the Ace lead, test in stageamt.wpengine.com (which must be kept up to date prod)
- Then it is implemented in Production and the asana tasks is marked “Published”
- The Ace Lead tests the feature in production and if passing marks the asana task “ready for documentation”
- The developer updates git and any other requested documentation and marks the task as Complete
- The Ace lead follows up to ensure the documentation is complete and supplements it as needed
Asana Available Status Tags
- For v#.# – indicates what version the feature or fix should be worked on in
- Approved for production – The Ace lead has approved implementation on the production version of the site
- Ready for documentation – The feature is implemented and tested in production and ready for documentation
Wireless hardware/access point credentials
- All FATT Access points should be remote accessible on the AMT Local wireless network specific to Access points (FATT).
- All Access point credentials should be stored in the LastPass vault associated with [email protected]
- User’s should never access that vault via the LastPass App but only use the browser
- To get the shared credentials to the vault please contact one of the software administrators (right now Crafty)
- All Credentials should be listed under the Asset number for the access point