Intelligent Build Manager

Welcome, New Build Manager!

2009-11-09T07:24:00.000-08:00

Hardware Environments

Web Servers (lms7fixws1/lmsenhws1)
File Servers (lmsfixfs/lmsenhfs)
Database Servers (lms7db)
Build Box (lms7build)
Subversion Server (subversion)
Cruise Control Server (vsafe2)

Application/Product Environments

LMS7
LAS (LASII)
LCMS
LUSAC
Ecommerce

Build Environments

lms7build: Automated Build Studio (pullit.bbp), in-house tools (vbs), stage files for each revision
lms7fixfs: Deploy tools, files for all revisions,builds, and pushes....need cleanup & backup
subversion: passwd files to control SVN commit (F:\SVN\REPOS\\conf\passwd)
vsafe2: watchout the over-growing log file (C:\build\server\ccnet.log)
krieweae7u: Earl's machine for ClearCase files (LCMS)

Build Process

Pull source files from source control server (update & compile locally)
Build package by revision range or cherry pick
Push code to INT by BM or hand them to TIS/DBA to deploy in QA/UAT/PROD
Email notification to related parties
Advance TTS in ClearQuest
Maintenance build environments after monthly builds

Tools

BuildManager.hta
- Code: hta
- Main GUI for Build Manager
- Server: lms7build
- Location: C:\_LoadBuild\Tools
PullIt.bbp

Code: Automated Build Studio file - need to be restarted every week to rlease memory
Server: lms7build
Location: C:\_LoadBuild\Tools\Auto_Build_Studio
Config file: :\_LoadBuild\Tools\Auto_Build_Studio\PullIt.ini

BuildIt.vbs
- Code: VBScript stand alone
- Server: lms7fixfs
- Location: F:\ACS\_LoadBuild\Tools
- Config File: F:\ACS\_LoadBuild\Tools\BuildIt.ini
PushIt.vbs

Code: VBScript stand alone
Server: lms7fixfs
Location: F:\ACS\_LoadBuild\Tools
Config File: F:\ACS\_LoadBuild\Tools\PushIt.ini

SendCompletedEmail.vbs

Code: VBScript stand alone
Server: lms7fixfs
Location: F:\ACS\_LoadBuild\Tools
Mailing List: F:\ACS\_LoadBuild\Tools\email.dat

CheckBuildProgress.vbs

Code: hta (with VBScript) file
Server: lms7build
Location: C:\Documents and Settings\hsiehla\My Documents\_BuildManager\Bin
Backend: ClearQuest server using lang's login criteria
MaintainUserPassword.wsf - to maintain password
CheckBuildProgress.bat - interface for BuildManager.hta

Pull_LAS
- Server: lms7build
- Location: C:\Documents and Settings\hsiehla\My Documents\_BuildManager\Bin
- Code: wsf file
- New architected build tools
  - wsf + VBScript: includes common function libraries
    - CDOEmail
    - ClearQuest
    - Database
    - Encryption
    - INI
    - OAM
    - RegularExpression
    - Subversion
    - XML
    - FileSystem
  - Structured folders
    - bin
    - conf
    - data
    - lib
    - lock
    - temp
    - work
  - Central INI file: C:\Documents and Settings\hsiehla\My Documents\_BuildManager\Conf\BuildManager.ini
  - Seperated xml file for each subsystem
    - BuildIt.xml
    - FolderStructure.xml
    - SQLDeployment.xml
Pull_HELP
Pull_LCMS
Build_LAS
Database Object Dependency
- Server: grahavie7u2003
- Location: C:\Documents and Settings\hsiehla\My Documents\_LoadBuild\Tools\SSIS\BuildManager\BuildManager.sln
- Requirement: SQL SQL Server Business Intelligence Development Studio (BIDS-2005 and above)
- Move to another server: Just Copy/Move over the BuildManager folder under SSIS to another machine with SQL BIDS installed
- Executions:
  - Double click on BuildManager.sln to open in BIDS
  - Run ReapplyExistingSP_CPE.dtsx under SSIS packages from Solution Explorer (help to rebuild object relationship in SQL due to we restore DB from PROD once a while) - about 15 min
  - Run ReapplyExistingSP_LMS.dtsx under SSIS packages from Solution Explorer (same as CPE) - about 40 min
  - Run FindDependency.dtsx under SSIS packages from Solution Explorer
  - Open DependencyResult_.txt file in Excel to view the result

Communications

Push: DBA + TIS + Tritschler + Mahagan
INT: all Dev + Tritschler + Mahagan
QA: all QA + all SM/PM + Tritschler + Mahagan + McBride + Martha
PROD: all QA + all SM/PM + Tritschler + Mahagan + McBride + Martha

Most Impact Areas - Dimensions

2009-10-26T07:46:00.000-07:00

Accuracy, Consistency, and Effectiveness

Load/Release Building

Update changed files only
Commits trigger the building process
Periodical/Scheduled builds
Deploy - real time or batch, self or hand over
MS Make (xml/response file)
Version the DLL files
Error Handling - Continue, Skip, or Stop
Send notification when failed to compile

Version Control

Access Control by ID/Branches
Branching by projects/functionality/bug ticket
Merging by dev/build
Repository Management

Automation
- Pull code from version control
- Make/Compile code
- Package
- Deploy Code/Data/SQL
- Notification
- Advance bug-tracking tickes
- Routine maintenance/Cleanup
- Tools/Data Backup

Integration
- Bug-Tracking Systems
- Versioning Control
- Deployment Verification
- Automatic QA
- Code Review
- Performance Tuning
- Deploy Code & SQL
- Central/Single GUI
- Push information/processes/work out from main stream to supported/integrated environments
- Pull information from data/status/work order files (xml)

Optimization
- Defect/bug-tracking ticke flow
- Build/Deploy flow
- Best practice for the company situation
- Naming convention - Files/Variables
- Folder Structures

Audit

What's in this Build list
What sliped in this Build list
Manually handled
Reports/Dashboards/Scorecards/Excell/Sharepoint/Powerpoint
Scope/Summary
Snapshot/Status
Alert
RedFlag

Communication
- Mailing lists for push/deploy/test events
- Content templates for different groups/roles
- Feedback
- 3-way handshaking protocol
- Realtime sync up (push) all parties by emails, IM, meetings, calls, dashboard, scorecard

Intelligence

Trend
Risk
Resource Management
From use cases to knowledge base and peer review feedback

Backup/Recovery
- Backup tools periodically
- Dump/backup repositories periodically
- Recovery plans/tools for rebuilding the environments in the case of machines/server crashed/moved/upgraded
- Recovery plans for all manual build in the case of emergency

Load Test

Issues are found after the push. Panic!?

2009-09-28T14:30:00.000-07:00

Few items we can take lessons from this senario:

Process: What is the standard procedures to follow?
Manage: Who should take ownership and drive the resolution?
Resource: Who should involve?
Documentation: What document is available?
Lessions: What we learned and how to prevent it in the future?

SQL Code (SP, UDF, TR) Guidelines

2009-09-23T14:05:00.000-07:00

Location
Coding
Naming Convention
Variables

Performance/Load Test Work List

2009-09-21T13:09:00.000-07:00

Challenge:
Load test is a cross function activity even we tend to make it a routine work. Because the Keynote account is maintain by the operation team, test users are controlled by DBA, and profiler capture is executed from the hosting company.

Build Status Audit Dashboard

2009-09-21T11:27:00.000-07:00

Challenge:
It is pretty common for QA manager, Project managers, and any upper managements to know the status of the next Build. So they can follow up on the progress, analyze the risk, and communicate to the clients accordingly. However, the scope of the build is dynamic. Not only those defect ticket's status are changing all the time along with the progress in multiple departments, the code and tickets might be pulled and included in the last minutes of everyday.

Load/Performance Test in QA before each Build

2009-09-14T08:54:00.000-07:00

7 highly effective habits

2009-09-14T07:45:00.000-07:00

Dependence to Independence
Habit 1: Be Proactive: Principles of Personal Choice
Habit 2: Begin with the End in Mind: Principles of Personal Vision
Habit 3: Put First Things First: Principles of Integrity & Execution

Independence to Interdependence
Habit 4: Think Win/Win: Principles of Mutual Benefit
Habit 5: Seek First to Understand, Then to be Understood: Principles of Mutual Understanding
Habit 6: Synergize: Principles of Creative Cooperation

Continual Improvement
Habit 7: Sharpen the Saw: Principles of Balanced Self-Renewal

BackUp - Quad II Task

2009-09-14T07:44:00.000-07:00

It is very important to backup your tools and data periodically starting from the very earily stage of the whole system development. Especially when you use Agile methods to expand your automation/integaration tools as they grow everyday. The automatical recovery system can wait but you just don't want to wait for the backup.

Here are some guidelines:

Similar folder structures on different servers and configuarable
PULL them to a central location
Log the backup activities (success & failure)
Backup to external devices (CDs, DVDs, external disks, etc.) periodically

Close to push date, PM wants to take out a bug-track ticket and revert the change

2009-09-14T07:08:00.000-07:00

Sometimes, PM (Project Manager) or AM (Account Manager) may want to change the scope of the build due to business reasons (i.e., contracts, payments, client budget change, negotiation gambits, etc.) at the last minutes. How can we as a IBM to build in the flexibilty in our system and process to serve business need?

Automation tests report failed result after a code push in QA,. Now what?

2009-09-10T07:15:00.000-07:00

I am sure you have simlar issues happening like this before. Usually, you can do the following:

Check what had been pushed and see if they are related to the failure
Check the old logs/results from the last automation test and confirm if they were passed
Manually reproduce the error from the front end and verify you get the same result
Code-wise, find out what was the last time they were modified
Push-wise, find out what is possible being changed in the environment
Data-wise, figure if it is code issue or a data issue

QA - Load/performance Test Plan

2009-06-26T11:49:00.000-07:00

The ultimate goal is to use load test to find out the performance bottle neck and eventually improve it. To find out the performance bottle neck, we would like to collect the real performance status on the production site then try to reproduce the issues/activities offline (in QA environment) and find out the root cause. Then we can do a before/after test to measure the performance improvement before push it to production.

For a web-based application, the performance might be affected by the following factors:

Network Bandwidth - This includes external (internet, traffic between hosting servers and the end users) and internal (intranet, traffic between file servers, DB servers, and Web servers)
Server Capacity - This includes all the capacity (CPU, memory, extra loads, etc) of Web servers, File servers, and Db servers
Application Efficiency - This includes front end (server side) and back end applications response time

Front End - This includes the efficiency of all server side pages, scripts, and pictures, and media files
Back End - This includes the efficiency of Database's SP, functions, and data's quality

So, before we spend the effort on any or all areas, let's find out where is the problem first. Like all business owners will like to find out who is their customer, what they want, what they buy, how they buy, and how much they buy, we would also like to find out those things from our customers so we can put those items they want on the most convenient places for them. In our case, if we know what 80% of our customers are doing, searching, and/or listing, then we will be able to improve the performance and improve 80% of the customer's satisfaction.

To illustrate this "Behavior Analysis", let's look at this commonly seen scenario in our application - Search and list records with multiple pages displayed.
Possible implementation:

Execute the query and put selected (usually, 10 records per page) records into a cursor. Because the cursor is existing in memory only, thus we need to execute the whole query every time. However, also because it is doing the transaction in memory, it will run faster.
Execute the query and put all selected records into a temp table. Then from here, we have the variance of using a temp table, common table expression, or global temp table.

Better implementation?
Rather than decide which one is more efficient, we want to know which one is more match what we need which is what the customer is doing. For example, if 80% of the transaction done by the user are just blindly search and then paging through to look for what they want, then obviously, how we handle the "paging" is more important than just try to optimize the core query they all execute. On the other hand, if most transactions show pretty much the search/list is done at the first 1-3 pages, then the process of getting that result set becomes critical. Additionally, if our database is suffered by "write" already, then we want to take the always-multiple-write from the first implementation into consideration. So again, if our profiler data tell us our SP spend a lot of time on "waiting" to read, then dump all result set to a session-based global temp table and read exclusively from there will give us a better result. And from here, goes on and on until we find a solution for our customer's pattern.

This technique for analyzing data is called "data mining", from whole bunch of data to find the pattern and then provide solution to business. Especially, we have seen/notified our issues must related by the behavior because it is always happening on the Monday morning. For no particular reasons, we would say our application should not act differently based on the time. It must be how the users using the application. But how? Let's find out.

First, we want to determine the bottle neck from the data, not just assumption. So we would like to capture and analyze the whole trip traffic from front end (the end user) to back end and then back to front end.

Second, We want to identify the transaction/activity happening during the peak hour. This is more like the shopping site want to know how far their potential customer go before they left the site. For them, they will then improve the "not-so-attractive" pages. For us, we want to know at which point they will do another activity (register, take courses, enter data, etc.). Once we have the traffic analysis, we can also provide it to our customer as the feedback of the usages/popularity so they can adjust their content, length, and interaction, etc.

Third, once the pattern is identified, we then can create/capture testing script and reproduce it in our non-production site for verification. When we know the pattern, we can then make business decision on how to implement the most impacted areas. Since our QA capacity is different than production, we can adjust the traffic accordingly but still based on the pattern.

Forth, once we have the proposed fix, we can test it in our controlable environment. So not only we want to fix the bottle neck caused by the pattern, we also want to prevent another pattern to cause the bottle neck. In this stage, we can increase/decrease the pressure on network traffic, DB response time, extra application occupied system resource on web servers, etc. Basically, after this, we know the fix can really fix the issue (how can we fix an issue without able to reproduce it?).

Fifth, the behavior/pattern will change. The tools and data we created need to be able to capture and analyze the future data to help us identify the new pattern. In the future, we just need to inject the data we captured from the troubled client within certain time frame. We can either give explanation to the client for the limitation of the application with solid analyzed data or we can tune our application from there.

Sixth, this technique/tools/methodology is reusable even we move on the next phase of ACS Learning. By being able to analyze "our" application, we can play a more professional role on servicing and retailing other vendor (SumTotal)'s product. Because we know our clients and we know our business.

Use Post-Commit Hook in Subversion for Email Notification

2009-06-25T07:20:00.000-07:00

Sector: Subversion (SVN) Management
Challenge: Instead of react to the request from development team to build and push the code, Intelligent Build Manager would like to monitor the commit activities and be notified by email automatically.
Client: Build manager
Solution:

Write a Post-Commit hook on Subversion server which will be triggered when a commit happened
Write a VBScript file to

Send email notification
Write a record to the BuildManager Database (development environment of our SSAS DW)

SVN Hooks:

The post-commit hook is invoked after a commit.
Subversion runs this hook by invoking a bat file under \SVN\REPOS\repository name\hooks\post-commit.bat with the following ordered arguments:

[1] REPOS-PATH (the path to this repository)
[2] REV (the number of the revision just committed)

The default working directory for the invocation is undefined, so the program should set one explicitly if it cares.
Because the commit has already completed and cannot be undone, the exit code of the hook program is ignored.
Note that 'post-commit' must be executable by the user(s) who will invoke it (typically the user httpd runs as), and that user must have filesystem-level permission to access the repository.
The hook program does not inherit the environment of its parent process. For example, a common problem is for the PATH environment variable to not be set to its usual value.

VB Scripting:

In post-commit.bat, you can do as simple as the following line to basically pass those 2 parameters:\_BuildManager\Bin\Notification_Commit.vbs "%1" "%2"
In Notification_Commit.vbs, use svnlook to get information then send emails through CDO object

Results:

Commit becomes a noticeable and traceable activity
While doing the email notification, this entry point also becomes a wonderful place to inject Subversion data to our BuildManager Data Warehouse. Save another ETL time and resources
Notification from "dev" branch becomes a proactive way to check progress and predict trunk commit and thus the progress of the working tickets.
By monitoring the flow from trunk commit to pull finish can usually help to detect the possible issues with pulling/compiling.

Lessions: Through multiple dimension approach, Build Manager finally can feel the move of the whole dynamic environment thus becomes a Intelligent Build Manager.

Why you need an Intelligent Build Manager

2009-06-22T08:30:00.000-07:00

See if this describe your situation:
You have been working hard to concentrate on building and developing your knowledge, product, and team for the past few years. And now it is the time to expand and grow the scale. Only there is a problem; the whole environment became complex and unmanageable. All of sudden, the multiple builds or deploys per day needs more human's attention, some last-minute fix need to be packaged or stripped out within limited time frame. You feel frustrated and need a dedicated person to take care the chaos and make it back to order.

However, all experienced technical staffs you have are working on something important. And most worsted, in deep down, they don't want to do this not-that-core-technology type of work. They are afraid to lose their state-of-the-art technical skills by doing this work over time. However, you know you need to find a technical enough and yet detail-oriented person who is willing to take it serious to build, coordinate, and manage the whole process and even develop some tools around it to make it automatic, mistaken-proven, and even traceable and analyzable.

Then when you look from the outside, you realize the candidates are either not technical enough to understand the technology you are developing and deploying or they are another highly focused technical talent who don't care about other part of business process.

It is at this moment, you need a partner, an Intelligent Build Manager, on your side to help you regain the control of your software development and management. Here are some challenges the Intelligent Build Manager can help you:

Understand the main technology and product of the development team and their various deployment needs
Setup, configuration, and integration of software development tools
Work closely with team leads from various departments to match the teams' styles and needs.
Automation tools for timely deploy, communication, account management
Define and improve process for developers and PM to follow and benefit from
Build environment management: accounts, branching, merging, repository, upgrade
Streamline the software development life cycle or Agile loop
Continuously integrate version control tools with defect/bug tracking tools, dynamic email distribution list, project management status reports, and QA needs
Develop, maintain, and manage Build related tools/programs for obvious reasons
Generate risk and resource analysis reports for senior management team to review and plan in the future

Why an "Intelligent" Build Manager?
You do actually need an intelligent individual to deal with those tasks not only that, but also you need the Business Intelligence (BI) technology to help your Build/Release Management. We want to apply BI's concept and tools to release management so we can do the data mining to find out how we can improve the application management processes. By using dashboard and scorecard, with the graphical KPI illustration, all involved parties can easily find the progress, forecast, and risk through the Web interface. This is the solution to fix the view difference between developers (tree) and project management (forest).

Please feel free to contact me if you have any questions or in need to resolve any or all of this issues.

Configurable Data

2009-06-19T08:55:00.000-07:00

Obviously, the more configurable data in your application, the more flexible this application can do for you.I will explain more in detail when talking about the multiple layers of the application

Manage SVN accounts through an user-friendly GUI

2009-06-19T07:08:00.000-07:00

Sector:Subversion (SVN) Management
Challenge: A systematic approach to manage, track, and communication of SVN account management.

Dev manager used to use the "walk-down-the-hall-to-talk" method to ask Build Manager for changes of the SVN account status. It is hard to track and the status is blind to all parties.
Build Manager used to manually modify the password file on SVN server for the change. It is time consuming. And when Build Manager was busy, sometimes he will even forget to do it or forget to notify. And once that was happening, it caused another round of task/resource spending on it again.
Developers was notified by either another "walk-to-talk" type of communication or Build Manager/Dev manager needs to manually write separate email to notify the developer of the change.

Client: Development Manager, Developers, VP of Deploy & Quality Control
Solution: A Process and a HTA program was created to streamline the task.

Developers will send a request email to Dev manager and copy Build manager
Dev manager will approve or reject the request by replying the email so Build Manager is also notified at the same time.
Build Manager then change the account status from the GUI and the tool will do the work behind the scene automatically including modify the password file, write a log, and send emails.
A notification email will be sent to the account owner, developer manager, Build Manager, and VP of Deploy & Quality control. And the email recipient list is configurable and coming from a central source ( See Integrate all ID systems to a central location post for more details).

Agile: SCRUM

The development of the software tool is on-the-go when all the status changes kept happening and people walking around with the old fashion way. Developers kept sending emails regarding the status of the account along with other regular and irregular code psuh and questions.
Status changes, GUI, notification, and procedures being scrummed out eventually.

Scripting: HTA - Select HTA as it has user-friendly GUI and easy to migrate to all web-based interface in the long run.
VBScript:

How to check a certain button is On or Off?
Read delimited data from a text file and parse them
Send email using CDO object
Make re-usable modules in VBScript with WSF file
Connect & retrieve Database through VBScript

Business Intelligence:

Account status log will be log into a Database table then eventually deploy to the Build Manager Datawarehouse cube for analysis
KPI:

Turnaround time for account status changes
Approval rate
Success rate for each commit/developer...when combine it with other data
Effectiveness of the changes, ie. numbers of TTS advanced by this change

Results:

All parties have a standard procedure to follow. One more step to improve the quality.
Improve 90% of the efficiency and correctness of SVN account management. Another candidate for cost reduction and revenue generating task.
Automatic email notification make it possible and easy to track. Prepare for a larger and more complex team and project.

Lessons: We are building the empire state building one brick at a time. This is the first step of the whole automation and effectiveness improving plan. Since it saves 90% of the time, I consider it as a quadrant-2 (important-but-not urgent) project. Glad I made the right decision on taking this automation.