VBScript Fundamentals for Windows Scripting – WMI

Before we begin, be forewarned that WMI encompasses A LOT of material and this will be a long article as is required to cover the material at even a high level. I will also preface the article with the fact that many of the diagrams and references included herein were derived from “The Windows Scripting Guide” by Microsoft Press which, in my opinion, is about the best book on Windows Scripting available at this time. Microsoft has done a great job in diagramming the many layers of the architecture that were used for modeling some of the diagrams seen later in this article. In addition, this article requires an intermediate to advanced knowledge of Widows Scripting and COM (Component Object Model) fundamentals. If you are not comfortable with the foundations of VBScript, please review these first two articles in this series:

VBScript Fundamentals for Windows Scripting – The Basics

VBScript Fundamentals for Windows Scripting – ADSI

In regards to managing resources on the network, you can’t find a more convenient tool than scripting with Windows Management Instrumentation, or WMI. WMI facilitates queries, monitoring, and changing settings on network resources including software, hardware, services, accounts, and many other objects. What used to require GUI interfaces to obtain information, such as how much physical memory was installed on a computer using System Properties, can now be done with one script launched from one machine against several remote machines.

The scripting syntax used with WMI is not quite as similar to that used in previous articles. It is still VBScript; however, the syntax used to connect to classes within WMI is more involved and looks more complicated. Don’t let that confuse you as you read. The majority of the code used in scripting WMI works as a template in that you can simply cut and paste the “meat and potatoes” of the code into almost every WMI script you write with minor editing to achieve the desired results. In fact Microsoft released a very cool tool called SCRIPTOMATIC that reveals the WMI classes on your machine as well as displays informational code pertaining to the WMI classes which you can modify directly or cut and paste into your own code.

Now, let’s dive into Windows Management Instrumentation beginning with a look at the WMI architecture itself.

Intro to WMI Architecture

The key to working with WMI is not actually writing the code, but finding out what classes are available to use within the script to access the required resources. For example, the Win32_Service class enables the collection of information about services, but how do find information on what other resource WMI can manage? Understanding the WMI architecture is the key to knowing what classes WMI can manage as well as what methods and properties can be used in each class. (if you need a refresher in properties and methods, please see my previous article here.)

WMI is made up of three main layers:

  1. Managed Resources
  2. WMI Infrastructure
  3. Consumers

These layers and their associated WMI components are displayed in the following diagram to demonstrate interoperability:

Figure

To obtain a better understanding, we will briefly look at the layers in reverse, beginning at the bottom as this is where the actual managed objects reside.

Managed Resources

This layer of the WMI model is quite easy to understand. It is simply the layer where items that you want to query or manage reside such as: disk drives, printers, registry, network interface cards, services, file system structures, hardware devices, etc. You can start to get an idea of how powerful WMI is by simply reading the incomplete list of resources WMI can manage in the previous sentence.

Let’s look at a simple WMI code to display the description of the loaded network adapters:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer)
Set colItems = objWMIService.InstancesOf("Win32_NetworkAdapterConfiguration")
For Each objItem in colItems
Wscript.Echo objItem.Description
Next

Depending on how many NIC’s you have installed and what types of settings are used in the network, you should receive an output similar to the following:

Figure

Note – To demonstrate the re-usability of the code, if we were to change Win32_NetworkAdapterConfiguration to read Win32_LogicalDisk and change the word Description to read DeviceID the output would list all the available drive letters on the machine.

WMI Providers

The Providers are one the main components of the core WMI architecture. Basically, the providers facilitate communication between the Consumer (the script itself) and the Managed Resources. The script example above uses the Win_32 Provider to obtain information about the network adapter as a managed resource. It is the Providers that allow the scripts to access the Win_32 API (Application Programming Interfaces) via scripting instead of needing a more robust programming language such as C++ or Visual Basic.

CIMOM

An acronym for Common Information Model Object Manager, the CIMOM facilitates interaction between Providers and Consumers. All requests sent from the WMI script to the Managed Resource pass through the CIMOM layer of the WMI architecture. It is the CIMOM layer that directs requests from the script to the appropriate Provider which then returns the requested information or performs the requested task. Let’s view this using a scripting example:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set colItems = objWMIService.InstancesOf("Win32_Process")
For Each objItem In colItems
Wscript.Echo objItem.Description
Next

In this script, we are returning a list of all running services on the local machine. CIMOM does not actually return the information, but rather forward the request to the Win32_Process Provider which actually does the legwork and then returns the information to the CIMOM layer of the architecture. CIMOM then forwards the returned information into the output you see as the results.

Note – You may notice that this script differs a little from the previous one in that the objWMIService variable has “\root\cimv2” appended to the end of the statement. This simply specifies the namespace that the query should be run against in the WMI Provider model. We will cover namespaces a little later. In this sample, you may remove the “\root\cimv2” and still obtain the same results providing you have not changed the default WMI namespace paths after installation.

CIM Repository

This is the object repository, or the schema, which defines all the data that is exposed by WMI. Think of the schema as a map that defines the objects stored in the repository. Using Active Directory as an example, First Name is an attribute of the User object. It is the schema that defines not only what type of information may be assigned to the First Name attribute, but also what the relationship is between the First Name attribute and the User object.

It is important to note here that while objects in the Active Directory are static for the most part, objects referenced in WMI are not. The state of the computer is dynamic and as such, a true repository, or snapshot, of the system is not possible. It is this design that can cause WMI scripts to perform slowly as they must retrieve new values each time they are run by sending requests to the CIM Repository. Querying or changing large amounts of information from several machines can take a while to complete using WMI due to the need to constantly obtain refreshed information.

The easiest way to learn about the classes, properties, and methods in the CIM repository and their relationships is to browse the repository itself. There are two tools that allow you to open the repository and visualize the interaction of these items. The WMI Tester is available as part of the default WMI install and can be located by searching for Wbemtest.exe on your machine. The second tool is called CIM Studio and may be obtained by downloading and installing the WMI Administration Tools here. Both are beneficial in examining the repository, but the CIM Studio utility offers a little more information and is a little easier to read.

Consumers

This is the highest layer in the Infrastructure model and the easiest to understand. In a nutshell, the Consumer is the script you write or the application you run that launches WMI code in the background.

CIM (Common Information Model)

This is the WMI schema that is used to store the class definitions that are used in modeling managed resources. It is the CIM definitions that allow us to make minor modifications to scripts and obtain completely different output as shown in the following two scripts:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer)
Set colItems = objWMIService.InstancesOf("Win32_OperatingSystem")
For Each objItem in colItems
Wscript.Echo "Name: " & objItem.Name
Wscript.Echo "BuildNumber: " & objItem.BuildNumber
Wscript.Echo "Manufacturer: " & objItem.Manufacturer
Next

The above script returns information regarding the Operating System currently installed on the local machine. Making some slight modifications to the script as shown below will return information about shares on the local machine.

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer)
Set colItems = objWMIService.InstancesOf("Win32_Share")
For Each objItem in colItems
Wscript.Echo "Description: " & objItem.Description
Wscript.Echo "Name: " & objItem.Name
Wscript.Echo "Path: " & objItem.Path
Next

The only changes made were to the class name (changing Win32_OperatingSystem to Win32_Share) and to the property values to return the appropriate information related to the different classes. The CIM allows us to obtain completely different results from a script without re-writing the entire script.

Note – If you are wondering how to figure out what properties are associated with what classes, it is here that the SCRIPTOMATIC tool comes into play. This tool enumerates every property for all available classes on the machines by simply selecting the appropriate class.

Namespaces

Namespaces are used to logically contain similar classes within the CIM. These are similar to folder path structures on a logical drive and follow the same rules in that while you cannot have two files with the same name in the same folder path, it is not possible to have two classes with the same name in a shared namespace. For a detailed view of the WMI Schema namespace, click here.

The default namespace is \root\cimv2 within WMI. This can be changed, but unless necessary for advanced scripting, it will usually be found within that context. You can find out what the current default namespace is on a machine by running the following script:

strComputer = "."

Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set colWMISettings = objWMIService.InstancesOf("Win32_WMISetting")

For Each objWMISetting in colWMISettings
Wscript.Echo "Default namespace for scripting: " & _
objWMISetting.ASPScriptDefaultNamespace
Next

Similarly, you can retrieve a list of all available namespaces by running the following script:

strComputer = "."
Call EnumNameSpaces("root")

Sub EnumNameSpaces(strNameSpace)
Wscript.Echo strNameSpace
Set objSWbemServices = _
GetObject("winmgmts:\\" & strComputer & "\" & strNameSpace)
Set colNameSpaces = objSWbemServices.InstancesOf("__NAMESPACE")
For Each objNameSpace In colNameSpaces
Call EnumNameSpaces(strNameSpace & "\" & objNameSpace.Name)
Next
End Sub

Running this using cscript from the command line returns the following output:

Figure

This script recursively searches through all available namespaces on the target computer (in this example, the local machine since we defined the strComputer variable as “.”) using a subroutine. When called, the subroutine uses the GetObject function to connect to the initial namespace identified as “root”, establishes a connection to the WMI provider, and then returns all instances of namespace beneath the current namespace using the For…Next control loop.

WMI Class Components

The CIM Repository contains several class categories. While the details of the categories are beyond the scope of this article, understanding the differences in the classes and the class types can be useful in determining classes to use in your scripting. More information on class categories and class types can be found here.

The WMI classes represent the templates mentioned earlier that define the manageability of the resource instances within WMI. Take for example the Win32_Process class. All processes in this class (which are the processes on the target machine) share the same properties which can be queried or managed using WMI. Classes represent tangible components on a computer which can be both hardware and/or software related (e.g. fixed disks, PCI cards, services, printers, network shares, etc.).

WMI Class Components – Properties

Properties are used to further describe the managed resources. For example, a NIC card has a description name, an OS installation has a build number, a monitor has a refresh rate, etc. In the following code snippet, the properties of the Win32_Process class are in bold:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set colItems = objWMIService.InstancesOf("Win32_Process")
For Each objItem In colItems
Wscript.Echo "Description: " & objItem.Description
Wscript.Echo "ExecutablePath: " & objItem.ExecutablePath
Wscript.Echo "Name: " & objItem.Name
Next

WMI Class Components – Methods

Methods are actions that can be taken directly upon a managed resource. Services present the best examples to demonstrate methods because of the actions that can be performed on them including stop, start, pause, install, delete, etc. In the following example we will SWbemObject in conjunction with a method to stop the Alerter service:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & _ \root\cimv2:Win32_Service.Name='Alerter'")
objWMIService.StopService

For this example the SWbemObject is actually a reference to objWMIService and the method used on the managed resource (the Alerter service) is StopService. Objects, such as the SWbemObject, declared in WMI are resources managed using the WMI Scripting Library object model which we’ll cover a little later.

WMI Class Components – Qualifiers

Qualifiers offer additional information about properties, classes, and methods. They define the operations of the called method, class or property. Using the example in the Properties section earlier, it is the qualifier that defines the properties used in the fact that the information retrieved is to be stored in an array within the control loop.

SWbemObject Concept Explained

To better understand SWbemObject, it helps to understand what an object path is. An object path is the logical path that defines an object’s location in the WMI Repository. The object path can be represented by the following:

[\\Computer Name] [\Namespace] [:Class Name] [.Property=Value]

Used in code, it would look like this:

… GetObject(“winmgmts:\\ServerA\root\cimv2:Win32_Service.Name=’Alerter’”)

The underlined portion in bold represents the object path defines as follows:

[\\Computer Name] = \\ServerA
[\Namespace] = \root\cimv2
[:Class Name] = :Win32_Service
[.Property=Value] = .Name=’Alerter’

To put it all together, consider the following script:

1. strComputer = "."

2. strNameSpace = "root\cimv2"

3. strClass = "Win32_Service"

4.

5. Set objClass = GetObject("winmgmts:\\" & strComputer & _

"\" & strNameSpace & ":" & strClass)

6.

7. For Each objClassProperty In objClass.Properties_

8.     Wscript.Echo objClassProperty.Name

9. Next

In the above script, lines 1-3 are initializing variables and assigning values, line 5 uses the GetObject function to connect to the WMI service and return the SWbemObject reference to the target class (in this example, the Win32_Service class), lines 7-9 used a For… Next control loop to display the properties collection for the Win32_Service class property. The output would resemble the following:

AcceptPause
AcceptStop
Caption
CheckPoint
CreationClassName
Description
DesktopInteract
DisplayName
ErrorControl
ExitCode
InstallDate
Name
PathName
ProcessId
ServiceSpecificExitCode
ServiceType
Started

Each of the above properties describe the Win32_Service class in that every service has a Description, a DisplayName, a Name, a value of Started (or stopped), etc…

To again illustrate the versatility and ease of use of the WMI code structure, if we were to use the same code as previously executed and simply change the three instances of the word “Properties” to the word “Methods” as seen here:

strComputer = "."

strNameSpace = "root\cimv2"

strClass = "Win32_Service"

Set objClass = GetObject("winmgmts:\\" & strComputer & _

"\" & strNameSpace & ":" & strClass)

For Each objClassMethod In objClass.Methods_

Wscript.Echo objClassMethod.Name

Next

We would return the following results using cscript from a command prompt:

Figure

As explained in the Methods section earlier, these are the actions that may be performed upon a managed resource within the Win32_Service class. In this instance, a managed resource might be the Alerter service, the Spooler service, the DHCP Server service, etc.
WMI Object Model

Now that we’ve touched on some of the WMI architecture, it is a good time to introduce the object model. Again, Microsoft has a nice flowchart of the library object model for WMI which can be found here. This will greatly aid in visualizing the process flow between the objects and instances mentioned so far. Take a few minutes to view the diagram and understand the processes occurring between objects in the library object model when different methods are called. This article will not go into details surrounding the object model; however, the following script will demonstrate the object calls made within the model.

strComputer = "ServerA"
Set objSWbemLocator = CreateObject("WbemScripting.SWbemLocator")
Set objSWbemServices = objSWbemLocator.ConnectServer _
(strComputer, "root\cimv2", "alan", "p@ssw0rd")
Set colSWbemObjectSet = objSWbemServices.InstancesOf("Win32_Process")
For Each objSWbemObject In colSWbemObjectSet
Wscript.Echo "Name: " & objSWbemObject.Name
Next

In the first line, we define the variable strComputer, next we create a reference to the SWbemLocator which resides at the top of the object model library (use the diagram for reference) using the CreateObject function. In the third line, we do two things: first we use the ConnectServer method of the SWbemLocator object to connect to the \root\cimv2 WMI namespace and second, pass user credentials to authenticate against the namespace on the remote server. This is only necessary if you are running the script in the context of a user account without administrative privileges on the remote computer. Now that we have connected to the WMI service, we again use the reference to connect to the Win32_Process class in line 4 (line 3 wraps around). Finally, we use the For… Next control loop in lines 5-7 to display the name property of each instance of the Win32_Process class.

For general WMI scripts used for tasks such as finding all the services on a machine, it is not completely necessary to understand every aspect of the WMI object model. However, a solid knowledge of the Automation objects (SWbem*) within the model is the beginning to writing more advanced WMI scripts.

Specifying Return Parameters in Queries

When using WMI to obtain information, it is not always desired to obtain all information regarding every instance of the object class being queried. Take the Event Logs for example. If you were to run the following script:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & _ "\root\cimv2")
Set colItems = objWMIService.InstancesOf("Win32_NTLogEvent")
For Each objItem in colItems
Wscript.Echo "EventCode: " & objItem.EventCode
Wscript.Echo "User: " & objItem.User
Wscript.Echo "Logfile: " & objItem.Logfile
Wscript.Echo "Message: " & objItem.Message
Next

You would receive information on every Event ID from every Event Log on the local system. While that is a lot of information, let’s take it a step further. Let’s say that every machine in an organization has a log file size limit of 30MB and that we use an array to query 50 different machines in one script. Since WMI scripts store their data in memory as they run, guess what’s going to occur. That’s right… system crash when we run out of available memory with the script process still hung running.

To narrow down the returned information in our script, we can use the WHERE clause in our variable statements. Up to now, we have been using the InstancesOf method in our script. To specify tighter query parameters, we will now use a new method called ExecQuery which replaces the InstancesOf method in the syntax. Compare the method below using our previous script with some modifications:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & _
"\root\cimv2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM " & _
"Win32_NTLogEvent WHERE Logfile = 'System'")
For Each objItem in colItems
Wscript.Echo "EventCode: " & objItem.EventCode
Wscript.Echo "User: " & objItem.User
Wscript.Echo "Logfile: " & objItem.Logfile
Wscript.Echo "Message: " & objItem.Message
Next

This would return specified information from the System Log only. Let’s examine the script more closely to identify the processes associated with the new ExecQuery method:

The first line sets the strComputer variable to the local machine. The second line (wrapped) uses the GetObject method to attach to the \root\cimv2 namespace on the local machine using the winmgmts:// WMI moniker (think of moniker as another word for name) and assign this value to the objWMIService variable. The third line (wrapped) uses the ExecQuery method of the SWbemObject (objWMIService) to specify the information query parameters. If you were reading the statement in plain English, it would read as follows, “Select all (*) from the Win32NTLogEvent class where the Logfile property equals System.” So basically, only information in the System log will be returned.

Finally we use the For… Next control loop to return specified property information on the objects return from the query.

We can even drill down further for more detailed returns with another slight modification to the ExecQuery clause as demonstrated below in the same script using bold text:

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & _
"\root\cimv2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM " & _
"Win32_NTLogEvent WHERE Logfile = 'System' AND " & _
"EventCode = ‘6005’")
For Each objItem in colItems
Wscript.Echo "EventCode: " & objItem.EventCode
Wscript.Echo "User: " & objItem.User
Wscript.Echo "Logfile: " & objItem.Logfile
Wscript.Echo "Message: " & objItem.Message
Next

This is the exact same script as before; however, using the AND clause we are now filtering the query for not only information specific to the System Event Log, but also only System Event Log information with the Event ID of 6005.

As you can see, using clauses can dramatically narrow your results and thereby limit the amount of data you would have to filter through to find the desired results. You may also notice that the query clauses resemble Structured Query Language (SQL). While the syntax is very similar, it is actually WMI Query Language (WQL) and not really SQL. Basically it doesn’t have the SQL functionality to UPDATE or INSERT or modify database tables as SQL does and is used for SELECT query statements.

Conclusion

You can definitely see how powerful WMI can be in Windows Scripting from the many previous examples. With a robust object model, access to an extensive amount of classes and methods, the ability to make very minor modifications to scripts to obtain completely different results, and the ability to easily specify finite target information in WMI queries it becomes apparent why more and more applications are leveraging WMI via scripting to retrieve information about the current state computing environment.

If you have experience with WMI, then hopefully this article clarified some behind the scenes processes that you may have had taken for granted when running your scripts. If you are new to WMI, don’t get discouraged. WMI is more in-depth than the other subjects we have covered in scripting so far and takes a little more effort to learn. Don’t let any confusion you may have now dissuade you from continuing to learn WMI as it can make your administration tasks immensely easier not to mention being an invaluable addition to your skill set. Take advantages of the many resources on WMI listed in this article to help and post any questions to the forums as you test your new WMI scripts.

The following are simply a few script examples that can be used as templates demonstrating a few common tasks that may be performed using WMI Scripting with minor editing changes. Once you grasp the WMI concepts, you will find you only need to write a few scripts to have a whole arsenal of powerful tools for administration.

Stopping Services

strComputer = "." ‘Edit this for remote machines
strNamespace = "\root\cimv2" ‘Edit to change namespace
strServiceName = "Alerter" ‘Edit to reflect service to be stopped

Set objWMIService = GetObject("winmgmts:" _
& "{impersonationLevel=impersonate}!\\" & strComputer & strNamespace)
Set colServices = objWMIService.ExecQuery _
("SELECT * FROM Win32_Service WHERE Name = '" & strServiceName & "'")
For Each objService in colServices
errReturnCode = objService.StopService() ‘StartService may also be used
Next

Forcing User Logoff

Const x = 4 'This can be changed to one of the following:
'0 = Logoff
'1 = Shutdown
'2 = Reboot
'4 = Forced Logoff
'5 = Forced Shutdown
'6 = Forced Reboot
'8 = Power Off Computer
'12 = Forced Power Off

strComputer = "." 'This can be changed for remote machines
strNamespace = "\root\cimv2" 'This can be changed for namespaces

Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" _
& strComputer & strNamespace)
Set colOperatingSystems = objWMIService.ExecQuery _
("SELECT * FROM Win32_OperatingSystem")
For Each objOperatingSystem in colOperatingSystems
ObjOperatingSystem.Win32Shutdown(x)
Next

Account Logon Failures

strComputer = "." ‘This can be changed for remote machines
strNamespace = “\root\cimv2\” ‘This can be changed for namespaces
strEventCode = “681” ‘Edit to reflect error code

Set objWMIService = GetObject("winmgmts:\\" & strComputer & _
"\root\cimv2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM " & _
"Win32_NTLogEvent WHERE Logfile = 'Security'AND " & _
"EventCode = ‘" & strEventCode & “’”)
For Each objItem in colItems
Wscript.Echo "EventCode: " & objItem.EventCode
Wscript.Echo "User: " & objItem.User
Wscript.Echo "Logfile: " & objItem.Logfile
Wscript.Echo "Message: " & objItem.Message
Next

Basic Linux Shell Scripting Part 3

In this 3rd and final article in my Shell Scripting series we will study, through example, methods of creating simple useful shell scripts. Some of the topics I will cover will be review of the previous two articles, others will cover some new things. The idea is to take what you have already learned and begin applying it to real world situations.

What do I create my scripts in?

Writing your own shell scripts requires you to know the very basic everyday Linux commands. For example, you should know how to copy, move, create new files, etc. The one thing you must know how to do is to use a text editor. There are three major text editors in Linux, vi, emacs and pico. If you are not familiar with vi or emacs, go for pico or some other easy to use text editor.

CAUTION: It is critical that you take care to not perform any scripting functions as the root user. To avoid this simply type the following commands:

Adduser scriptuser
Passwd scriptuser
su scriptuser

This will create a user that you can dedicate to running scripts.

Your First BASH Program

Our first program will be the classical “Hello World” program. Yes, if you have programmed before, you must be sick of this by now. However, this is traditional, and who am I to change tradition? The “Hello World” program simply prints the words “Hello World” to the screen. So fire up your text editor, and type the following inside it:

#!/bin/bash
echo "Hello World"

The first line tells Linux to use the bash interpreter to run this script. In this case, bash is in the /bin directory. If bash is in a different directory on your system, make the appropriate changes to the line. Explicitly specifying the interpreter is very important, so be sure you do it as it tells Linux which interpreter to use to run the instructions in the script. The next thing to do is to save the script. We will call it hello.sh. With that done, you need to make the script executable:

shell$ chmod 700 ./hello.sh

shell$ ./hello.sh
Hello World

There it is! Your first program! Boring and useless as it is, this is how everyone starts out. Just remember the process here. Write the code, save the file, and make it executable with chmod.

A More Useful Program

We will write a program that will move all files into a directory, and then delete the directory along with its contents, and then recreate the directory. This can be done with the following commands:

shell$ mkdir trash
shell$ mv * trash
shell$ rm -rf trash
shell$ mkdir trash

Instead of having to type all that interactively on the shell, write a shell program instead:

#!/bin/bash
mkdir trash
mv * trash
rm -rf trash
mkdir trash
echo "Deleted all files!"

Save it as clean.sh and now all you have to do is to run clean.sh and it moves all files to a directory, deletes them, recreates the directory, and even prints a message telling you that it successfully deleted all files. So remember, if you find that you are doing something that takes a while to type over and over again, consider automating it with a shell program. You may notice that this is very similar to batch file programming under DOS. In fact, UNIX shells scripts are the grandfather of Batch files.

Comments

Comments help to make your code more readable. They do not affect the output of your program. They are specially made for you to read. All comments in bash begin with the hash symbol: “#”, except for the first line (#!/bin/bash). The first line is not a comment. Any lines after the first line that begin with a “#” is a comment. Take the following piece of code:

#!/bin/bash
# this program counts from 1 to 10:
for i in 1 2 3 4 5 6 7 8 9 10; do
echo $i
done

Even if you do not know bash scripting, you immediately know what the above program does, because of the comment. It is good practice to make use of comments. You will find that if you need to maintain your programs in the future, having comments will make things easier.

Variables

Variables are basically “boxes” that hold values. You will want to create variables for many reasons. You will need it to hold user input, arguments, or numerical values. Take for instance the following piece of code:

#!/bin/bash
x=12
echo "The value of variable x is $x"

What you have done here, is to give x the value of 12. The line echo “The value of variable x is $x” prints the current value of x. When you define a variable, it must not have any whitespace in between the assignment operator: “=”. Here is the syntax:

variable_name=this_value

The values of variables can be accessed by prefixing the variable name with a dollar symbol: “$”. As in the above, we access the value of x by using echo $x.

There are two types of variables. Local variables, and environmental variables. Environmental variables are set by the system and can usually be found by using the env command. Environmental variables hold special values. For instance, if you type:

shell$ echo $SHELL
/bin/bash

You get the name of the shell you are currently running. Environmental variables are defined in /etc/profile and ~/.bash_profile. The echo command is good for checking the current value of a variable, environmental, or local. If you are still having problems understanding why we need variables, here is a good example:

#!/bin/bash
echo "The value of x is 12."
echo "I have 12 pencils."
echo "He told me that the value of x is 12."
echo "I am 12 years old."
echo "How come the value of x is 12?"

Okay, now suppose you decide that you want the value of x to be 8 instead of 12. What do you do? You have to change all the lines of code where it says that x is 12. But wait… there are other lines of code with the number 12. Should you change those too? No, because they are not associated with x. Confusing right? Now, here is the same example, only it is using variables:

#!/bin/bash
x=12 # assign the value 12 to variable x
echo "The value of x is $x."
echo "I have 12 pencils."
echo "He told me that the value of x is $x."
echo "I am 12 years old."
echo "How come the value of x is $x?"

Here, we see that $x will print the current value of variable x, which is 12. So now, if you wanted to change the value of x to 8, all you have to do, is to change the line x=12 to x=8, and the program will automatically change all the lines with $x to show 8, instead of 12. The other lines will be unaffected. Variables have other important uses as well, as you will see later on.

Control Structures

Control structures allow your program to make decisions and to make them more compact. More importantly as well, it allows us to check for errors. So far, all we have done is write programs that start from the top, and go all the way to the bottom until there are no more commands left in the program to run. For instance:

#!/bin/bash
cp /etc/foo .
echo "Done."

This little shell program, call it bar.sh, copies a file called /etc/foo into the current directory and prints “Done” to the screen. This program will work, under one condition. You must have a file called /etc/foo. Otherwise here is what happens:

shell$ ./bar.sh
cp: /etc/foo: No such file or directory
Done.

So you can see, there is a problem. Not everyone who runs your program will have /etc/foo in their system. It would perhaps be better if your program checked if /etc/foo existed, and then if it did, it would proceed with the copying, otherwise, it would quit. In pseudo code, this is what it would look like:

if /etc/code exists, then
copy /etc/code to the current directory
print "Done." to the screen.
otherwise,
print "This file does not exist." to the screen
exit

Can this be done in bash? Of course! The collection of bash control structures are, if, while, until, for and case. Each structure is paired, meaning it starts with a starting “tag” and ends with an ending “tag”. For instance, the if structure starts with if, and ends with fi. Control structures are not programs found in your system. They are a built in feature of bash. Meaning that from here on, you will be writing your own code, and not just embedding programs into your shell program.

if … else … elif … fi

One of the most common structures is the if structure. This allows your program to make decisions, like, “do this if this conditions exists, else, do something else”. To use the if structure effectively, we must make use of the test command. test checks for conditions, that is, existing files, permissions, or similarities and differences. Here is a rewrite on bar.sh:

#!/bin/bash
if test -f /etc/foo
then
# file exists, so copy and print a message.
cp /etc/foo .
echo "Done."
else
# file does NOT exist, so we print a message and exit.
echo "This file does not exist."
exit
fi

Notice how we indent lines after then and else. Indenting is optional, but it makes reading the code much easier in a sense that we know which lines are executed under which condition. Now run the program. If you have /etc/foo, then it will copy the file, otherwise, it will print an error message. test checks to see if the file /etc/foo exists. The -f checks to see if the argument is a regular file. Here is a list of test’s options:

-d check if the file is a directory
-e check if the file exists
-f check if the file is a regular file
-g check if the file has SGID permissions
-r check if the file is readable
-s check if the file’s size is not 0
-u check if the file has SUID permissions
-w check if the file is writeable
-x check if the file is executable

else is used when you want your program to do something else if the first condition is not met. There is also the elif which can be used in place of another if within the if. Basically elif stands for “else if”. You use it when the first condition is not met, and you want to test another condition.

If you find that you are uncomfortable with the format of the if and test structure, that is:

if test -f /etc/foo
then

then, you can do it like this:

if [ -f /etc/foo ]; then

The square brackets form test. If you have experience in C programming, this syntax might be more comfortable. Notice that there has to be white space surrounding both square brackets. The semicolon: “;” tells the shell that this is the end of the command. Anything after the semicolon will be run as though it is on a separate line. This makes it easier to read basically, and is of course, optional. If you prefer, just put then on the next line.

When using variables with test, it is a good idea to have them surrounded with quotation marks. Example:

if [ “$name” -eq 5 ]; then

while … do … done

The while structure is a looping structure. Basically what it does is, “while this condition is true, do this until the condition is no longer true”. Let us look at an example:

#!/bin/bash
while true; do
echo "Press CTRL-C to quit."
done

true is actually a program. What this program does is continuously loop over and over without stopping. Using true is considered to be slow because your shell program has to call it up first and then run it. You can use an alternative, the “:” command:

#!/bin/bash
while :; do
echo "Press CTRL-C to quit."
done

This achieves the exact same thing, but is faster because it is a built in feature in bash. The only difference is you sacrifice readability for speed. Use whichever one you feel more comfortable with. Here is perhaps, a much more useful example, using variables:

#!/bin/bash
x=0; # initialize x to 0
while [ "$x" -le 10 ]; do
echo "Current value of x: $x"
# increment the value of x:
x=$(expr $x + 1)
sleep 1
done

As you can see, we are making use of the test (in its square bracket form) here to check the condition of the variable x. The option -le checks to see if x is less than, or equal to the value 10. In English, the code above says, “While x is less than 10 or equal to 10, print the current value of x, and then add 1 to the current value of x.”. sleep 1 is just to get the program to pause for one second. You can remove it if you want. As you can see, what we were doing here was testing for equality. Check if a variable equals a certain value, and if it does, act accordingly. Here is a list of equality tests:

Checks equality between numbers:
x -eq y Check is x is equals to y
x -ne y Check if x is not equals to y
x -gt y Check ifx is greater than y
x -lt y Check if x is less than y

Checks equality between strings:
x = y Check if x is the same as y
x != y Check if x is not the same as y
-n x Evaluates to true if x is not null
-z x Evaluates to true if x is null.

The above looping script we wrote should not be hard to understand, except maybe for this line:

x=$(expr $x + 1)

The comment above it tells us that it increments x by 1. But what does $(…) mean? Is it a variable? No. In fact, it is a way of telling the shell that you want to run the command expr $x + 1, and assign its result to x. Any command enclosed in $(…) will be run:

#!/bin/bash
me=$(whoami)
echo "I am $me."

Try it and you will understand what I mean. The above code could have been written as follows with equivalent results:

#!/bin/bash
echo "I am $(whoami)."

You decide which one is easier for you to read. There is another way to run commands or to give variables the result of a command. This will be explained later on. For now, use $(…).

until … do … done

The until structure is very similar to the while structure. The only difference is that the condition is reversed. The while structure loops while the condition is true. The until structure loops until the condition is true. So basically it is “until this condition is true, do this”. Here is an example:

#!/bin/bash
x=0
until [ "$x" -ge 10 ]; do
echo "Current value of x: $x"
x=$(expr $x + 1)
sleep 1
done

This piece of code may look familiar. Try it out and see what it does. Basically, until will continue looping until x is either greater than, or equal to 10. When it reaches the value 10, the loop will stop. Therefore, the last value printed for x will be 9.

for … in … do … done

The for structure is used when you are looping through a range of variables. For instance, you can write up a small program that prints 10 dots each second:

#!/bin/bash
echo -n "Checking system for errors"
for dots in 1 2 3 4 5 6 7 8 9 10; do
echo -n "."
done
echo "System clean."

In case you do not know, the -n option to echo prevents a new line from automatically being added. Try it once with the -n option, and then once without to see what I mean. The variable dots loops through values 1 to 10, and prints a dot at each value. Try this example to see what I mean by the variable looping through the values:

#!/bin/bash
for x in paper pencil pen; do
echo "The value of variable x is: $x"
sleep 1
done

When you run the program, you see that x will first hold the value paper, and then it will go to the next value, pencil, and then to the next value, pen. When it finds no more values, the loop ends.

Here is a much more useful example. The following program adds a .html extension to all files in the current directory:

#!/bin/bash
for file in *; do
echo "Adding .html extension to $file..."
mv $file $file.html
sleep 1
done

If you do not know, * is a wild card character. It means, “everything in the current directory”, which is in this case, all the files in the current directory. All files in the current directory are then given a .html extension. Recall that variable file will loop through all the values, in this case, the files in the current directory. mv is then used to rename the value of variable file with a .html extension.

case … in … esac

The case structure is very similar to the if structure. Basically it is great for times where there are a lot of conditions to be checked, and you do not want to have to use if over and over again. Take the following piece of code:

#!/bin/bash
x=5 # initialize x to 5
# now check the value of x:
case $x in
0) echo "Value of x is 0."
;;
5) echo "Value of x is 5."
;;
9) echo "Value of x is 9."
;;
*) echo "Unrecognized value."
esac

The case structure will check the value of x against 3 possibilities. In this case, it will first check if x has the value of 0, and then check if the value is 5, and then check if the value is 9. Finally, if all the checks fail, it will produce a message, “Unrecognized value.”. Remember that “*” means “everything”, and in this case, “any other value other than what was specified”. If x holds any other value other than 0, 5, or 9, then this value falls into the *’s category. When using case, each condition must be ended with two semicolons. Why bother using case when you can use if? Here is the equivalent program, written with if. See which one is faster to write, and easier to read:

#!/bin/bash
x=5 # initialize x to 5
if [ "$x" -eq 0 ]; then
echo "Value of x is 0."
elif [ "$x" -eq 5 ]; then
echo "Value of x is 5."
elif [ "$x" -eq 9 ]; then
echo "Value of x is 9."
else
echo "Unrecognized value."
fi

Quotations

Quotation marks play a big part in shell scripting. There are three types of quotation marks. They are the double quote: “, the forward quote: ‘, and the back quote: `. Does each of them mean something? Yes.

The double quote is used mainly to hold a string of words and preserve whitespace. For instance, “This string contains whitespace.”. A string enclosed in double quotes is treated as one argument. For instance, take the following examples:

shell$ mkdir hello world
shell$ ls -F
hello/ world/

Here we created two directories. mkdir took the strings hello and world as two arguments, and thus created two directories. Now, what happens when you do this:

shell$ mkdir "hello world"
shell$ ls -F
hello/ hello world/ world/

It created a directory with two words. The quotation marks made two words, into one argument. Without the quotation marks, mkdir would think that hello was the first argument, and world, the second.

Forward quotes are used primarily to deal with variables. If a variable is enclosed in double quotes, its value will be evaluated. If it is enclosed in forward quotes, its value will not be evaluated. To make this clearer, try the following example:

#!/bin/bash
x=5 # initialize x to 5
# use double quotes
echo "Using double quotes, the value of x is: $x"
# use forward quotes
echo 'Using forward quotes, the value of x is: $x'

See the difference? You can use double quotes if you do not plan on using variables for the string you wish to enclose. In case you are wondering, yes, forward quotes can be used to preserve whitespace just like double quotes:

shell$ mkdir 'hello world'
shell$ ls -F
hello world/

Back quotes are completely different from double and forward quotes. They are not used to preserve whitespace. If you recall, earlier on, we used this line:

x=$(expr $x + 1)

As you already know, the result of the command expr $x + 1 is assigned to variable x. The exact same result can be achieved with back quotes:

x=`expr $x + 1`

Which one should you use? Whichever one you prefer. You will find the back quote used more often than the $(…). However, I find $(…) easier to read, especially if you have something like this:

$!/bin/bash
echo "I am `whoami`"

Arithmetic with BASH

BASH allows you to perform arithmetic expressions. As you have already seen, arithmetic is performed using the expr command. However, this, like the true command, is considered to be slow. The reason is that in order to run true and expr, the shell has to start them up. A better way is to use a built in shell feature which is quicker. So an alternative to true, as we have also seen, is the “:” command. An alternative to using expr, is to enclose the arithmetic operation inside $((…)). This is different from $(…). The number of brackets will tell you that. Let us try it:

#!/bin/bash
x=8 # initialize x to 8
y=4 # initialize y to 4
# now we assign the sum of x and y to z:
z=$(($x + $y))
echo "The sum of $x + $y is $z"

As always, whichever one you choose, is purely up to you. If you feel more comfortable using expr to $((…)), by all means, use it.

bash is able to perform, addition, subtraction, multiplication, division, and modulus. Each action has an operator that corresponds to it:

ACTION OPERATOR
Addition +
Subtraction –
Multiplication *
Division /
Modulus %

Everyone should be familiar with the first four operations. If you do not know what modulus is, it is the value of the remainder when two values are divided. Here is an example of arithmetic in bash:

#!/bin/bash
x=5 # initialize x to 5
y=3 # initialize y to 3

add=$(($x + $y)) # add the values of x and y and assign it to variable add
sub=$(($x - $y)) # subtract the values of x and y and assign it to variable sub
mul=$(($x * $y)) # multiply the values of x and y and assign it to variable mul
div=$(($x / $y)) # divide the values of x and y and assign it to variable div
mod=$(($x % $y)) # get the remainder of x / y and assign it to variable mod

# print out the answers:
echo "Sum: $add"
echo "Difference: $sub"
echo "Product: $mul"
echo "Quotient: $div"
echo "Remainder: $mod"

Again, the above code could have been done with expr. For instance, instead of add=$(($x + $y)), you could have used add=$(expr $x + $y), or, add=`expr $x + $y`.

Reading User Input

Now we come to the fun part. You can make your program so that it will interact with the user, and the user can interact with it. The command to get input from the user, is read. read is a built in bash command that needs to make use of variables, as you will see:

#!/bin/bash
# gets the name of the user and prints a greeting
echo -n "Enter your name: "
read user_name
echo "Hello $user_name!"

The variable here is user_name. Of course you could have called it anything you like. read will wait for the user to enter something and then press ENTER. If the user does not enter anything, read will continue to wait until the ENTER key is pressed. If ENTER is pressed without entering anything, read will execute the next line of code. Try it. Here is the same example, only this time we check to see if the user enters something:

#!/bin/bash
# gets the name of the user and prints a greeting
echo -n "Enter your name: "
read user_name

# the user did not enter anything:
if [ -z “$user_name” ]; then
echo “You did not tell me your name!”
exit
fi

echo “Hello $user_name!”

Here, if the user presses the ENTER key without typing anything, our program will complain and exit. Otherwise, it will print the greeting. Getting user input is useful for interactive programs that require the user to enter certain things. For instance, you could create a simple database and have the user enter things in it to be added to your database.

Functions

Functions make scripts easier to maintain. Basically it breaks up the program into smaller pieces. A function performs an action defined by you, and it can return a value if you wish. Before I continue, here is an example of a shell program using a function:

#!/bin/bash
# function hello() just prints a message
hello()
{
echo "You are in function hello()"
}

echo “Calling function hello()…”
# call the hello() function:
hello
echo “You are now out of function hello()”

Try running the above. The function hello() has only one purpose here, and that is, to print a message. Functions can of course be made to do more complicated tasks. In the above, we called the hello() function by name by using the line:

hello

When this line is executed, bash searches the script for the line hello(). It finds it right at the top, and executes its contents.

Functions are always called by their function name, as we have seen in the above. When writing a function, you can either start with function_name(), as we did in the above, or if you want to make it more explicit, you can use the function function_name(). Here is an alternative way to write function hello():

function hello()
{
echo "You are in function hello()"
}

Functions always have an empty start and closing brackets: “()”, followed by a starting brace and an ending brace: “{…}”. These braces mark the start and end of the function. Any code enclosed within the braces will be executed and will belong only to the function. Functions should always be defined before they are called. Let us look at the above program again, only this time we call the function before it is defined:

#!/bin/bash
echo "Calling function hello()..."
# call the hello() function:
hello
echo "You are now out of function hello()"

# function hello() just prints a message
hello()
{
echo “You are in function hello()”
}

Here is what we get when we try to run it:

shell$ ./hello.sh
Calling function hello()...
./hello.sh: hello: command not found
You are now out of function hello()

As you can see, we get an error. Therefore, always have your functions at the start of your code, or at least, before you call the function. Here is another example of using functions:

#!/bin/bash
# admin.sh - administrative tool

# function new_user() creates a new user account
new_user()
{
echo “Preparing to add a new user…”
sleep 2
adduser # run the adduser program
}

echo “1. Add user”
echo “2. Exit”

echo “Enter your choice: ”
read choice

case $choice in
1) new_user # call the new_user() function
;;
*) exit
;;
esac

In order for this to work properly, you will need to be the root user, since adduser is a program only root can run. Hopefully this example (short as it is) shows the usefulness of functions.

Trapping

You can use the built in command trap to trap signals in your programs. It is a good way to gracefully exit a program. For instance, if you have a program running, hitting CTRL-C will send the program an interrupt signal, which will kill the program. trap will allow you to capture this signal, and will give you a chance to either continue with the program, or to tell the user that the program is quitting. trap uses the following syntax:

trap action signal

action is what you want to do when the signal is activated, and signal is the signal to look for. A list of signals can be found by using the command trap -l. When using signals in your shell programs, omit the first three letters of the signal, usually SIG. For instance, the interrupt signal is SIGINT. In your shell programs, just use INT. You can also use the signal number that comes beside the signal name. For instance, the numerical signal value of SIGINTis 2. Try out the following program:

#!/bin/bash
# using the trap command

# trap CTRL-C and execute the sorry() function:
trap sorry INT

# function sorry() prints a message
sorry()
{
echo “I’m sorry Dave. I can’t do that.”
sleep 3
}

# count down from 10 to 1:
for i in 10 9 8 7 6 5 4 3 2 1; do
echo $i seconds until system failure.”
sleep 1
done
echo “System failure.”

Now, while the program is running and counting down, hit CTRL-C. This will send an interrupt signal to the program. However, the signal will be caught by the trap command, which will in turn execute the sorry() function. You can have trap ignore the signal by having “”” in place of the action. You can reset the trap by using a dash: “-“. For instance:

# execute the sorry() function if SIGINT is caught:
trap sorry INT

# reset the trap:
trap – INT

# do nothing when SIGINT is caught:
trap ” INT

When you reset a trap, it defaults to its original action, which is, to interrupt the program and kill it. When you set it to do nothing, it does just that. Nothing. The program will continue to run, ignoring the signal.

AND & OR

We have seen the use of control structures, and how useful they are. There are two extra things that can be added. The AND: “&&” and the OR “||” statements. The AND statement looks like this:

condition_1 && condition_2

The AND statement first checks the leftmost condition. If it is true, then it checks the second condition. If it is true, then the rest of the code is executed. If condition_1 returns false, then condition_2 will not be executed. In other words:

if condition_1 is true, AND if condition_2 is true, then…

Here is an example making use of the AND statement:

#!/bin/bash
x=5
y=10
if [ "$x" -eq 5 ] && [ "$y" -eq 10 ]; then
echo "Both conditions are true."
else
echo "The conditions are not true."
fi

Here, we find that x and y both hold the values we are checking for, and so the conditions are true. If you were to change the value of x=5 to x=12, and then re-run the program, you would find that the condition is now false.

The OR statement is used in a similar way. The only difference is that it checks if the leftmost statement is false. If it is, then it goes on to the next statement, and the next:

condition_1 || condition_2

In pseudo code, this would translate to the following:

if condition_1 is true, OR if condition_2 is true, then…

Therefore, any subsequent code will be executed, provided at least one of the tested conditions is true:

#!/bin/bash
x=3
y=2
if [ "$x" -eq 5 ] || [ "$y" -eq 2 ]; then
echo "One of the conditions is true."
else
echo "None of the conditions are true."
fi

Here, you will see that one of the conditions is true. However, change the value of y and re-run the program. You will see that none of the conditions are true.

If you think about it, the if structure can be used in place of AND and OR, however, it would require nesting the if statements. Nesting means having an if structure within another if structure. Nesting is also possible with other control structures of course. Here is an example of a nested if structure, which is an equivalent of our previous AND code:

#!/bin/bash
x=5
y=10
if [ "$x" -eq 5 ]; then
if [ "$y" -eq 10 ]; then
echo "Both conditions are true."
else
echo "The conditions are not true."
fi
fi

This achieves the same purpose as using the AND statement. It is much harder to read, and takes much longer to write. Save yourself the trouble and use the AND and OR statements.

Using Arguments

You may have noticed that most programs in Linux are not interactive. You are required to type arguments, otherwise, you get a “usage” message. Take the more command for instance. If you do not type a filename after it, it will respond with a “usage” message. It is possible to have your shell program work on arguments. For this, you need to know the “$#” variable. This variable stands for the total number of arguments passed to the program. For instance, if you run a program as follows:

shell$ foo argument

$# would have a value of one, because there is only one argument passed to the program. If you have two arguments, then $# would have a value of two. In addition to this, each word on the command line, that is, the program’s name (in this case foo), and the argument, can be referred to as variables within the shell program. foo would be $0. argument would be $1. You can have up to 9 variables, from $0 (which is the program name), followed by $1 to $9 for each argument. Let us see this in action:

#!/bin/bash
# prints out the first argument
# first check if there is an argument:
if [ "$#" -ne 1 ]; then
echo "usage: $0 "
fi

echo “The argument is $1”

This program expects one, and only one, argument in order to run the program. If you type less than one argument, or more than one, the program will print the usage message. Otherwise, if there is an argument passed to the program, the shell program will print out the argument you passed. Recall that $0 is the program’s name. This is why it is used in the “usage” message. The last line makes use of $1. Recall that $1 holds the value of the argument that is passed to the program.

Redirection and Piping

As we covered before, redirection and piping are very common. Here are some examples and a little review in the context of usable scripts.

shell$ echo "Hello World"
Hello World

Redirection allows you to redirect the output somewhere else, most likely, a file. The “>” operator is used to redirect output.

shell$ echo "Hello World" > foo.file
shell$ cat foo.file
Hello World

Here, the output of echo “Hello World” is redirected to a file called foo.file. When we read the contents of the file, we see the output there. There is one problem with the “>” operator. It will overwrite the contents of any file. What if you want to append to it? Then you must use the append operator: “>>”. It is used in the exact same was as the redirection operator, except that it will not overwrite the contents of the file, but add to it.

Piping allows you to take the output from a program, and then run the output through another program. Piping is done using the pipe operator: “|”.:

shell$ cat /etc/passwd | grep shell
shell:x:1002:100:X_console,,,:/home/shell:/bin/bash

Here we read the entire /etc/passwd and then pipe the output to grep, which in turn, searches for the string shell and then prints the entire line containing the string, to the screen. You could have mixed this with redirection to save the final output to a file:

shell$ cat /etc/passwd | grep shell > foo.file
shell$ cat foo.file
shell:x:1002:100:X_console,,,:/home/shell:/bin/bash

It worked. /etc/passwd is read, and then the entire output is piped into grep to search for the string shell. The final output is then redirected and saved into foo.file. You will find redirection and piping to be useful tools when you write your shell programs.

Temporary Files

Temporary files are useful when you want to store information, or persist information to disk for a short period of time. A temporary file can be created with the $$ symbol. This symbol uses a random number generator to either prefix or suffix the file and ensure it has a unique name.

shell$ touch hello
shell$ ls
hello
shell$ touch hello.$$
shell$ ls
hello hello.689

There it is, your temporary file

Return Values

Most programs return a value depending upon how they exit. For instance, if you look at the manual page for grep, it tells us that grep will return a 0 if a match was found, and a 1 if no match was found. Why do we care about the return value of a program? For various reasons. Let us say that you want to check if a particular user exists on the system. One way to do this would be to grep the user’s name in the /etc/passwd file. Let us say the user’s name is foobar:

shell$ grep "foobar" /etc/passwd
shell$

No output. That means that grep could not find a match. But it would be so much more helpful if a message saying that it could not find a match was printed. This is when you will need to capture the return value of the program. A special variable holds the return value of a program. This variable is $?. Take a look at the following piece of code:

#!/bin/bash
# grep for user foobar and pipe all output to /dev/null:
grep "foobar" /etc/passwd > /dev/null 2>&1
# capture the return value and act accordingly:
if [ "$?" -eq 0 ]; then
echo "Match found."
exit
else
echo "No match found."
fi

Now when you run the program, it will capture the return value of grep. If it equals to 0, then a match was found and the appropriate message is printed. Otherwise, it will print that there was no match found. This is a very basic use of getting a return value from a program. As you continue practicing, you will find that there will be times when you need the return value of a program to do what you want.

If you happen to be wondering what 2>&1 means, it is quite simple. Under Linux, these numbers are file descriptors. 0 is standard input (eg: keyboard), 1 is standard output (eg: monitor) and 2 is standard error (eg: monitor). All normal information is sent to file descriptor 1, and any errors are sent to 2. If you do not want to have the error messages pop up, then you simply redirect it to /dev/null. Note that this will not stop information from being sent to standard output. For example, if you do not have permissions to read another user’s directory, you will not be able to list its contents:

shell$ ls /root
ls: /root: Permission denied
shell$ ls /root 2> /dev/null
shell$

As you can see, the error was not printed out this time. The same applies for other programs and for file descriptor 1. If you do not want to see the normal output of a program, that is, you want it to run silently, you can redirect it to /dev/null. Now if you do not want to see either standard input or error, then you do it this way:

shell$ ls /root > /dev/null 2>&1

You should recall that this is IO Redirection, and the null device represents nothing or nowhere.

Now what if you want your shell script to return a value upon exiting? The exit command takes one argument. A number to return. Normally the number 0 is used to denote a successful exit, no errors occurred. Anything higher or lower than 0 normally means an error has occurred. This is for you, the programmer to decide. Let us look at this program:

#!/bin/bash
if [ -f "/etc/passwd" ]; then
echo "Password file exists."
exit 0
else
echo "No such file."
exit 1
fi

By specifying return values upon exit, other shell scripts you write making use of this script will be able to capture its return value. This is similar to programming with functions in other languages.

Porting BASH Scripts

The most well written scripts are portable. That means that they work on many different version of Linux and Unix with little modification. The biggest mistake you can make when considering portability is to use applications that only exist on a single version of Linux. The foo program is an example. It servers the same purpose as echo, but does not exist on all flavors of Linux.

Basic Linux Shell Scripting Part 2

This article will focus on an important part of Shell Scripting – Redirection and Piping. Simply put, redirection allows you to use the output of one command as the input for another. This allows you to perform tasks such as keyword searching a directory listing, or making decisions based on the content of a file. There are a few basic things you must understand to perform redirection.

Standard Input

Standard Input or [stdin] refers to the place where input for a command usually comes, you keyboard. Unless you specify otherwise, input always comes from standard input. Standard input is redirected using the < symbol.

Standard Output

Line standard input, standard output is the place where, unless modified, the output of commands is sent. This is by default your console, tty, or vty session. To send the output of a command elsewhere, you can use the > or >> symbols. Note that the >> symbol will append data, while the > symbol will overwrite what is already there.

Standard Error

Standard error is the place where errors are displayed for the user to review. This also defaults to your current display session. To record the errors of a command, you can use the 2> symbol. This can be useful if you want to run a command and keep a log of all errors the command generates.

Consider this example of redirection. The first command generates a list of all files on the computer, redirecting the contents to the file dirlist.out and the errors to the file dirlist.err. It is also set to run as a background process, allowing us to continue to work. During it’s execution I ran [ps] twice. The first time you can see the job running, the second one indicates its completion. Then I made a typo. Finally is a list of the two files used in the first command.

Piping

Piping allows you to take the output of one command and use it as the input to another. This can be used to “glue” commands together to perform combined operations. You have already done this to some extent when you have run the [ls | more] command to page through longer directory listings. The pipe operator is the [|] symbol, usually located above backslash “\”. This symbol performs a dual redirection, redirecting the output of the command on the left side of the symbol to the input for the command on the right side. Consider this example.

Regular Expressions

Regular expressions were once the primary way that searches were performed in programming languages. They are common in languages such as C and PERL. Regular expressions are becoming increasingly common in modern programming languages such as Java and Visual Basic.NET. Modern Regular Expressions are even standardized in ISO/IEC 9945-2:1993.

Regular Expressions: Regular expressions are a context-independent syntax that can represent a wide variety of character sets and character set orderings, where these character sets are interpreted according to the current locale.

Mastering regular expressions is a worthwhile goal if you plan on working extensively with UNIX, or programming in any modern language. There have been many books published on this topic, with focus books on the various regular expression packages such as those that come with Java, Perl (perlre), Visual Basic.NET and C#.

The GREP utility is produced by GNU and it’s primary function is to search through input for defined expressions. When using the GREP utility, and expression is encased within two forward slashes, //. The values between these symbols represent the search argument. In it’s simplest form, a regular expression could be /findme/. This would locate the word “findme”. It’s not until we begin adding wildcard symbols and logic expression that the regular expression begins gain power.

Consider the following characters.
1. . (period) – means any single character in this position
2. * – 0 or more occurrences of the preceding character
3. [ ] – a range of acceptable characters
4. [^ ] – a range of unacceptable characters
5. ^ – characters must be at the beginning of the line
6. $ – characters must be at the end of the line
7. \< \> – indicated that the characters are treated as a word

Consider the following examples.
1. d.g will find dog but not doug.
2. B* will find all words with the letter B.
3. [df].g will find dog and fog, but not bog.
4. [^df].g will find log and bog, but not dog and fog.
5. ^ef will find effort but not referee.
6. $in will find bin but not inside
7. [^\] will not find concatenate.

Using GREP

One of the most powerful commands you can have in your Linux command arsenal is GREP. GREP supports a number of input parameters, each of which can be used to alter the method of search. The man page and the –help argument can be used to obtain detailed description of each option. GREP is most often used as the target of a pipe from another command. Here is an example of GREP being used to parse through a file to locate a particular line. This example locates all commands the history commands that contain [ls].

Here are some common questions and answers about grep usage.**

1. How can I list just the names of matching files?

grep -l ‘main’ *.c

lists the names of all C files in the current directory whose contents mention `main’.

2. How do I search directories recursively?

grep -r ‘hello’ /home/gigi

searches for `hello’ in all files under the directory `/home/gigi’. For more control of which files are searched, use find, grep and xargs. For example, the following command searches only C files:

find /home/gigi -name ‘*.c’ -print | xargs grep ‘hello’ /dev/null

This differs from the command:

grep -r ‘hello’ *.c

which merely looks for `hello’ in all files in the current directory whose names end in `.c’. Here the `-r’ is probably unnecessary, as recursion occurs only in the unlikely event that one of `.c’ files is a directory.

3. What if a pattern has a leading `-‘?

grep -e ‘–cut here–‘ *

searches for all lines matching `–cut here–‘. Without `-e’, grep would attempt to parse `–cut here–‘ as a list of options.

4. Suppose I want to search for a whole word, not a part of a word?

grep -w ‘hello’ *

searches only for instances of `hello’ that are entire words; it does not match `Othello’. For more control, use `\<' and `\>‘ to match the start and end of words. For example:

grep ‘hello\>’ *

searches only for words ending in `hello’, so it matches the word `Othello’.

5. How do I output context around the matching lines?

grep -C 2 ‘hello’ *

prints two lines of context around each matching line.

6. How do I force grep to print the name of the file?

Append `/dev/null’:

grep ‘eli’ /etc/passwd /dev/null

gets you:

/etc/passwd:eli:DNGUTF58.IMe.:98:11:Eli Smith:/home/do/eli:/bin/bash

7. Why do people use strange regular expressions on ps output?

ps -ef | grep ‘[c]ron’

If the pattern had been written without the square brackets, it would have matched not only the ps output line for cron, but also the ps output line for grep. Note that some platforms ps limit the ouput to the width of the screen, grep does not have any limit on the length of a line except the available memory.

8. Why does grep report “Binary file matches”?

If grep listed all matching “lines” from a binary file, it would probably generate output that is not useful, and it might even muck up your display. So GNU grep suppresses output from files that appear to be binary files. To force GNU grep to output lines even from files that appear to be binary, use the `-a’ or `–binary-files=text’ option. To eliminate the “Binary file matches” messages, use the `-I’ or `–binary-files=without-match’ option.

9. Why doesn’t `grep -lv’ print nonmatching file names?

`grep -lv’ lists the names of all files containing one or more lines that do not match. To list the names of all files that contain no matching lines, use the `-L’ or `–files-without-match’ option.

10. I can do OR with `|’, but what about AND?

grep ‘paul’ /etc/motd | grep ‘franc,ois’

finds all lines that contain both `paul’ and `franc,ois’.

11. How can I search in both standard input and in files?

Use the special file name `-‘:

cat /etc/passwd | grep ‘alain’ – /etc/motd

12. How to express palindromes in a regular expression?

It can be done by using the back referecences, for example a palindrome of 4 chararcters can be written in BRE.

grep -w -e ‘\(.\)\(.\).\2\1’ file

It matches the word “radar” or “civic”.

Guglielmo Bondioni proposed a single RE that finds all the palindromes up to 19 characters long.

egrep -e ‘^(.?)(.?)(.?)(.?)(.?)(.?)(.?)(.?)(.?).?\9\8\7\6\5\4\3\2\1$’ file

Note this is done by using GNU ERE extensions, it might not be portable on other greps.

13. Why are my expressions whith the vertical bar fail?

/bin/echo “ba” | egrep ‘(a)\1|(b)\1’

The first alternate branch fails then the first group was not in the match this will make the second alternate branch fails. For example, “aaba” will match, the first group participate in the match and can be reuse in the second branch.

14. What do grep, fgrep, egrep stand for ?

grep comes from the way line editing was done on Unix. For example, ed uses this syntax to print a list of matching lines on the screen.

global/regular expression/print

g/re/p

fgrep stands for Fixed grep, egrep Extended grep.

** Source: http://www.gnu.org/software/grep/doc/grep_13.html#SEC1

Conclusion

So as you can probably imagine, what you can do with tools such as GREP, Input and Output redirection and Piping is only limited by your creativity and knowledge of the system you are working on. In my next article we will take our basic shell scripts, as well as these commands and begin creating some functional administrative scripts.

** Source: http://www.gnu.org/software/grep/doc/grep_13.html#SEC1

Visual Basic Loops

This article, we will be dealing with ways to have your program perform the same task a number (not necessarily known at the time you start programming) of times.

For… Next…

Using the For.. Next statement, you can set code to run a predefined number of times. As an example, let’s build a small application that asks the user for a number. The application will then add up all the even numbers from 0 to that number (or all odd numbers if that’s what the user wants).

We start by creating the interface. Add the following controls to your form.

We use the frame to more or less group the two optionbuttons. That way, when a user sees the application, it will immediately be clear that the two optionbuttons are ‘grouped’, and that it’s only possible to select one of the two. (As an added bonus, VB will recognize the OptionButtons are grouped, and if one is selected, the other will automatically be deselected).

You do this by first drawing the frame on the form, and then drawing the OptionButtons in the Frame. The end result should look something like this.

Ok, so we have the interface. Let’s spend a minute thinking on what the Program is supposed to do. The user will enter a number, select either odd or even, and our program will add all the numbers from 0 to the specified number. The way we will handle this is by using the For.. Next statement. First of all, let’s look at the syntax for the statement:

For counter = start To end [Step step]

[statements]

Next [counter]

Basically, you specify a counter (usually a variable), and you specify the inital (start) value, as well as the end value. The code to be executed every time is specified in between the ‘For’ and the ‘Next’ statement (The [statements] in the above quote). The Step-part can be used in case you want to skip a certain number of items in between. (Note: You can also use a negative number here).

If we look back at our program, if a user selects ‘Even’, we will calculate all values from 0 to whatever number was entered. The tricky part is that not all numbers are odd. We could code a way to determine whether or not a number is odd or even, but why not keep it more simple than that? We know that every other number will be even. So we just start at 0, skip the odd number 1 (which we will achieve by setting the ‘Step’ part of the For… Next statement to 2), and add the number to another variable.

If the user picked odd, we start at 1 instead of at 0, and also always skip a number. That’s basically it.

In code this would look something like this:

Private Sub cmdCalculate_Click()
‘ Declare variables
Dim intRange As Integer
Dim intResult As Long

‘ Check whether we need to calculare odd or even numbers
‘ Even was selected
If optEven.Value = True Then ‘ Even
For intRange = 0 To Val(txtNumber.Text) Step 2
intResult = intResult + intRange
Next intRange
‘ Odd was Selected
Else
For intRange = 1 To Val(txtNumber.Text) Step 2
intResult = intResult + intRange
Next intRange
End If

MsgBox “Result:” & intResult
End Sub

Let’s look at what we did here. First we declare two variables, intRange, which will be the number used to ‘remember’ where we are when we loop between 0 (or 1) and the number specified by the user. We’ve made this variable an integer, which (as you know) limits the range of the to anything between -32,678 and 32,676. The other variable was made a Long. This is done because obviously the number will de bigger than the number contained in the integer.

Next, we check which OptionButton was selected. If it was the Even one (which is the default), we start intRange at 0. Else, we start at 1. (This is nice about using two OptionButtons. If the first one is NOT true, then the other must be).

Then comes the For Next statement:

For intRange = 0 To Val(txtNumber.Text) Step 2
intResult = intResult + intRange
Next intRange

We specify that intRange will contain anything between 0 and the specified number. This is where the program starts looping. First it sets intRange to 0, and runs the piece of code inbetween the For and the Next statements (in our case: intResult = intResult + intRange. It’s not hard to see what we do here, is it? We just use intResult to store the value in between. We just add the (at that moment) current value of intRange to it).

Finally, the Next intRange statement is called. To see this in a bit more detail, let’s look at all the things that would happen if we’d set the number to 10, and want all the even numbers added up.

# intRange will be set to run from 0 to 10.
# The loop begins (No, not a giggle loop). intRange is, at this point 0. intResult is also 0.
# 0 + 0 = 0, so intResult remains 0 for the moment. (You might find this a bit redundant, but from a mathematical perspective, this is the proper way to tackle this problem, I think).
# Next is called. intRage is now set to 2 (The step-part was set to 2, remember?).
# 0 + 2 = 2. intResult = 2.
# Next is called. intRange is set to 4.
# 2 + 4 = 6.
# Next is called. intRange is set to 6.
# 6 + 6 = 12.
# Next is called. intRange is set to 8.
# 12 + 8 = 20.
# Next is called. intRange is set to 10. (Not that 10 is the final value in the range. The For… Next-loop will end after this).
# 20 + 10 = 30.
# The user sees a MessageBox which displays the result

It is possible to exit the loop in between, by using the ‘Exit For’ statement. (If, for instance, you’d be trying to find a certain condition or number, and the number would be reached, you can prevent your program from running longer than necessary by using Exit For).

That’s it for For… Next. Let’s now take a look at Do-loops.

Do… Loop

Another way to deal with code that has to be used several times in a row is by using a Do… Loop-statement. You can choose from Do – While or Do – Until. Both accept a condition. While the condition is not true( when you use Until), the loop continues. Lets look at another example:

Dim x As Integer
Do Until x = 10
x = x + 1
Loop

MsgBox x

We could rewrite this code to make it into a Do… While statement like this:

Dim x As Integer

Do While x < 10 x = x + 1 Loop MsgBox x As you see, the While form is slightly different. While the condition is true, the loop continues. So what's the difference between Do... Loop and For... Next loops? Well, with For... Next, you usually know how many times something has to be looped, while using Do... Loop usually means you just know that a certain condition needs to be True.

Basic Linux Shell Scripting Concepts

If you have worked with any flavor of Unix you have no doubt encountered the shell script. Shell scripts are one of the most important aspects of a Unix system. In fact, virtually all operations that take place on a Unix computer are accomplished via shell scripts. Shell scripts can range in complexity from simple to massive. In my installation article you encountered the ./configure and ./make scripts. If you use VI to review the source of these scripts, it can be overwhelming. In this article I will unravel some of the basics of shell scripting. This is part 1 of a three part series on shell scripting. The next article will address manipulation of command input and output.

What is a Script?

A script is a series of statements that are used to construct an operation. Scripts are often used to automate repetitive tasks and to simplify complex ones. One of the biggest advantages that Unix has traditionally held over systems such as Windows and Netware is its ability to automate tasks without the need for complex programming languages. That is changing however. On Windows, the Windows Script Host (WSH) is used to provide command line access to languages such as VBScript, and Jscript, as well as a host of others. This replaces the primitive batch file mechanism that many people are familiar with. Netware provides a powerful logon script language. Regardless of the source of implementation, scripts generally provide similar functionality. Below is a list of common features found in script engines:

Temporary storage of information: Maintain state of counters, names, etc. Usually accomplished through variables.

Conditional Logic: Allows you to change the path of execution based on changing conditions and values. This is accomplished via IF statements and LOOP constructs.

Arrays: Tables of data based in memory.

Access to key operating system functions: Allows the manipulation of the computer and its contents outside of the scope of the file system. On Windows this is done via ADSI (Active Directory Scripting Interface) and Network and File System objects. On Unix everything is in the file system, so this is done by reading from and writing to files.

Command Line Execution: Administrators can run scripts by executing a command at a prompt. In Windows this is done via WSH and on Unix by the Execute (x) bit.

Visual Basic Arrays

Welcome back to our series of articles about Visual Basic programming. Last article explained the basics of using variables and constants, which were (as we saw) little areas of memory in which we could store values, in order to quickly perform alterations on the values, or just temporarily maintain values, in order to make sure our Form would not be completely filled with TextBoxes that had data in them.

This week, we will be looking at a different type of variables, namely arrays. I know I promised last week to also tackle procedures, but I’ve decided against that, seeing that the discussion of arrays took a little more space than I intended in the first place. Also, I don’t want to create confusion by linking arrays to procedures. I’ll tackle procedures next week. That’s also when we’ll add some functionality to our calculator project.

Arrays

All right. We saw we use variables to store data in. We could store the name of a book, we could store a date, numbers, etc. What though if we wanted to store two values that are related? For instance, suppose we want to make a list of friend’s birthdays. If we’d do that in Word or HTML, we’d most probably use a table, and would come up with something like this.

Yes, these are their actual birth dates, and yes, programmers get to hang with the cool people 🙂 . Don’t forget to send them a card!

Anyway, as you can see, we’re talking two dimensional data here. It’s not just a name, and it’s not just a date; it’s a combination of both. We could store those values in separate strings and separate data-variables, but we’d end up with a whole lot of them, and it would be rather difficult to see which birth date belongs to which person. Visual Basic luckily offers another way to store two- (or more) dimensional data in: arrays .

Ok, so how would we go about using an array to store the list of friends and their birthdays, as shown above? Simple. We declare an array just the same way as we would declare any variable. The only difference is that we will specify exactly how many dimensions our array will have. In the above example, we have two columns, and five rows. The statement to declare an array for this would be:

Dim strFriends(5, 2) as String

I chose for string here, because of the date. If we’d try and add a date in the format we used (01/01/2002), because of the slash (/), Visual Basic would think we’d want to divide the numbers

The dimensions of an array are quite simple. It might take a little effort to get used to it, but if you’ve dealt with matrices before, this ought to be a rather easy approach. We’ll take the above example again, but this time I’ll show the dimensions next to it.

If we want to set or retrieve data from an array, we always start with the row number, and then use the column number (Same as in Excel. In excel, you deal with cell A1 first). In our example, the name ‘Gary Oldman’ has the ‘coordinates’ (For lack of a better word) 5, 1. Bill Clinton’s birthday’s coordinates are 2, 2.

Let’s put the data from the birthday table into an array in VB. Just start a new EXE-project.

Right under ‘Option Explicit’, we’ll declare the variable:

Dim strFriends(5, 2) As String

Go Back to the Form (double-click it’s name in the Project Explorer), and double-click on the Form. We’ll be taken into the Form_Load event. (This is a From’s basic event, and will be triggered when we actually start a project. As soon as the form becomes loaded, the event fires).

Once we have declared the variable, we can start adding data to it:

strFriends(1, 1) = “Alicia Silverstone”
strFriends(1, 2) = “10/4/1976”

strFriends(2, 1) = “Bill Clinton”
strFriends(2, 2) = “08/19/1946”

strFriends(3, 1) = “Bill Gates”
strFriends(3, 2) = “10/28/1955”

strFriends(4, 1) = “Claudia Schiffer”
strFriends(4, 2) = “08/25/1970”

strFriends(5, 1) = “Gary Oldman”
strFriends(5, 2) = “03/21/1958”

Ok, so now we defined the array, and stored our little friends list in it. But what use is that really, unless we can actually retrieve the values in there? Let’s extend our Form a little. Add the following controls to your Form.

Notice I didn’t name the labels the proper way. Labels are not likely to be addressed in code. I know it’s not conform the standard I brought up a little while ago, but personally, I only use label-naming when I know I’ll change it’s value from code. Also, the TabIndex property might need a little explanation. TabIndex is the order in which you can use the TAB-key to move from one control to another. The control with the lowest TabIndex will first receive focus when the program loads. In other words: In this example, if we start our program, the cursor will be in the txtRow-TextBox. If you’d press TAB, it would move to txtColumns, and lastly to the CommandButton. The last TextBox (txtValue) also has a few new properties. BackColor we saw before. Enabled means that a user can click the Box, and see the cursor move into there. If ‘Enabled’ is ‘False’, this is not possible. ‘Locked’ means a user can’t type data into the TextBox. I combined these three here, to achieve the effect that the boxes’ appearance indicates that it’s only purpose is to provide output, and not input. Lastly, the ‘Default’ property for the CommandButton indicates that this button will activate when the user presses the Enter key. Since this is the only button, we set Default to True. Your form now ought to look similar to this.

See how the bottom TextBox appears to say: “You can’t enter data into me. I’m just meant to present output to you”? Anyway, double-click on the CommandButton, to get us into the Click Event for the button. There, we add the following code:

txtValue.Text = strFriends(Val(txtRow), Val(txtColumn))

If you run the program now, you’ll see that the TabIndex works wonderfully. Also, if you use the Enter key, it’s as if you clicked the button; the Click event will be fired. Enter a number between 1 and 5 in the first box, and 1 or 2 in the second one, and the bottom TextBox will show you the result. Notice that if you use ‘bigger’ coordinates, you will receive an error. That’s because the array is only 5 by 2 for now.

The form of arrays we discussed today is known as ‘Static Arrays’. I.e. we define beforehand exactly how big the array will be. There’s other forms of arrays as well, namely ‘Dynamic Arrays’ (which the dimensions of are not known during the coding), and Control Arrays ( which are arrays of Active-X Controls). We’ll discuss Dynamic Arrays in a future article.

Control Arrays

A Control Array is an array of Active-X Controls. Well, that doesn’t help much, does it? A Control Array exists of a number of the same Controls (For instance buttons) that share the same name, and events. Ah, that’s a lot better, Peter. So what’s the difference between them? Simple. Like the ‘coordinates’ in the array we used, Control Arrays also have dimensions. Controls in a Control Array can be uniquely identified by their ‘Index’-number.

So why would we want to use Control Arrays? Simple. Two reasons:

# Control Arrays are less memory-intensive than the same number of regular controls.
# Control Arrays give you the ability to write one shared Event for all of the controls. This

means you have to write code once in order to respond to the same event for all the buttons (or other control-types) in the Control Array.

In order to create a Control Array, you can do two things:

# Add a control to the Form. Add another control of the same type, and give it the exact same name as the first Control.
# Add a control to your Form. Use Copy (CTRL + C), and then Paste (CTRL + V).

Both of the times you’ll receive a question if you want to create a Control Array. Next week, we’ll make a Control Array out of the buttons on our calculator, and the whole concept might get a bit clearer.

Visual Basic Variables and Constants

In the last article, we took a look at what the term ‘event-driven’ means. We saw that Visual Basic will only process code after a certain event takes place, and we saw a few tips that can make your programming-life easier. We also started off with a very basic calculator program.

This article, I’ll explain what variables are, why and how to use them, what different types of variables you can use, the scope of variables and finally we’ll look at constants.

Variables described

What are variables? According to the MSDN, a variable is A location in the computer’s memory where data is stored. Anybody got that? Let’s try clarifying that a little. Think of a variable as a wallet; a place where you temporarily store various things in (coins, bills, credit cards, pictures, business cards). Whenever you need one of those things, you retrieve it from the wallet, and when you’re finished with it, you can put it back in.

The same can happen with data in applications. Last article, we did not do anything with the number that was provided by the user. We just left it in the TextBox. We could have stored it in our computer’s memory, and cleared the TextBox. Another way to use variables would be by storing text that’s provided by a user.

Why use variables?

Good question. Why not just use the TextBoxes to store the data in? Well, look back at the calculator. If we’d store all numbers provided by the user in text boxes, we’d soon have a screen full of text boxes with various numbers in it. Instead, in the future, we will store the number in a variable and then clear the TextBox (thus indicating to the user that our program is ready to receive a new number).

Another reason to use variables would be because the memory is readily available for fast access. If something is stored in memory, it can be called or altered with great speed. If we’d just store our data in text boxes, the program would first have to read the number in the TextBox, then alter it, and finally put it back into the TextBox (Granted, this might seem like a marginal fraction of time, but in programming, you’re ideally looking for code that’s as optimized as can be. In the end, every millisecond you can spare counts).

Finally, Visual Basic offers a number of functions that can be used to alter data. Using variables, you can easily store the data, and use those functions on it.

How to use variables

Before you can use a variable, you have to declare it. That is, you have to let Windows know you want to reserve memory for storing data in, supply a name for the variable, and you have to tell Windows what type of data it will be (discussed in the variable-types-section). The way to do this is by using the keyword Dim. (Dim stems from earlier days, and is an abbreviation for Dimension, because you reserved a dimension of the memory. If you like, think of Dim as meaning ‘Declare In Memory’, which is a nice little shortcut that makes sure you’ll remember the ‘Dim’-keyword for now. (In a few weeks, you won’t even have to think about using Dim, it’ll be an automatic thing)).

To declare a variable that we’ll be using to store text in (A so-called string), we would use the following piece of code (the name of the variable would be strInput):

Dim strInput As String

This is called Explicit Declaration. Remember that when we installed VB, we checked the ‘Require Variable Declaration’ CheckBox? (This causes ‘Option Explicit’ to be written in the code window). Well, by checking that box, we made sure that every variable we will be using will have to be declared with the Dim statement. This ensures that if you try assigning data to, or calling data from a variable, Visual Basic will generate an error if the name of the variable doesn’t exist. When you create a variable, Visual Basic will (silently) create the variable with value ‘0’.

If you want to assign a value to a declared variable, you can do that the following way:

strInput = txtInput.Text or
strInput = “Hello!”

The text “Hello” is between quotes (“) because it’s a string. We have to let Visual Basic know exactly what text we want to store in the string, because the internal commands, functions, and keywords of Visual Basic could also be used within a string. Consider the following (try it out in VB if you want. You’ll receive an error):

Dim strText As String
strText = This is an example of how to use the keyword Dim in VB

See? Visual Basic has no idea of whether it would have to use Dim to declare something, or whether you were trying to store it within a string. The code will work if we add the quotes:

Dim strText As String
strText = “This is an example of how to use the keyword Dim in VB”

The quotes only have to be used in the case of strings. All the other variable-types require no quotes to be used.

Types of variables

Because there are various data-types to store, there are also various variable-types. The table below shows respectively the types of variables, the number of memory-bytes required to store the variables in, and the range of the values for each variable-type.

As you can see, any possible value you would ever want to store has been thought of. The default type for a variable, by the way, is Variant. So if you’d just declare a variable like this:

Dim strInput

The type would be Variant (which takes up most memory). Try to save a copy of the table somewhere close by. Try and always pick the variable type that would require the least bytes of memory to be reserved. Remember, the more efficient you use the memory, the faster your application will run.

Scope of variables

We saw that variables are declared with the ‘Dim’-keyword. If you use Dim, a variable will only exist within the procedure in which it was declared. (As an example, if we’d declare a variable within a button-click event, as soon as all code within that event is processed, the variable will cease to exist, and the data inside the variable will be lost).

Hmmm. What if we want to use a variable to store a number for a bit longer? (Think back to our calculator. If the variable would lose it’s data after the click event was processed, we wouldn’t have much use for it, would we?). Well, it’s possible to declare a variable that is created right when the program starts, and will exist as long as the program is running. A variable like that is called a static variable. That kind of variable is declared by using the ‘Static’ keyword:

Static strInput as String

This variable will remain active within the procedure it’s created in (Again, a single click -event for example). Lastly, it is also possible to declare a variable that’s valid for the whole application. Those are called public variables, and are accessible from anywhere within the application. They’re declared with the Public keyword:

Public strInput as String

A Public variable can only be declared at the top-level of the code (Under ‘Option Explicit’). If you try and use it anywhere else (for example within a click-event) you will receive an error if you try to run the application. Same as with picking the right variable-type, always pick the most limited scope. This prevents memory from being used unnecessary, and mistakes from happening.

Constants

Well, as we know now, variables are used for temporarily storing data. What though if we would have a variable that stays the same within the entire lifespan of the application, and is used all-through the application? (Think of a filename that is not going to change within the program, and on which multiple procedures rely). Well, this is also possible. We can use a specific type of variable called a Constant to achieve exactly this.

A constant is declare like so:

Const cstrFilename As String = “C:\autoexec.bat”

As you see, we immediately assign the value of the constant to it. This is because the value will be constant. It will not change. The c is to show this is a Constant. This makes it easier for you to recognise the Constant within the application.

Your First Visual Basic Program

Last week we took a look at some of the standard controls that come with Visual Basic. We saw that we can change the way a control looks, by fiddling around with it’s properties.

This week, we will look at some other possibilities of controls; events. We will build a very basic program, and I’ll offer some practical tips in the process.

We will build a little calculator during this article. Fire up Visual Basic, and select ‘Standard Exe’ from the templates. You’ll notice that Visual Basic supplied some very basic and vague names for the form and the project name. (How could it not? VB doesn’t know what we will be working on). Since we want to be able to recognize the project in the future by a useful name, we’ll change the project’s name first.

In the Project Explorer Window, click on Project 1. (The uppermost-item, or ‘root-item’). In the Properties Window, you will see that ‘name’ is the only property of the project we can change. We’ll set it to ‘Calculator’ for now. As soon as you press Enter, you will notice that in the Form Designer, the new name will be reflected.

Next, we will change the Form’s name and Caption to something a little more descriptive. Click on the Form (Either in the Project Explorer, or in the Form Designer), and change it’s Caption to ‘Calculator’. We’ll set the name of the form to frmCalculator.

Hold it. Stop right there. frmCalculator? Has Peter finally managed to go mad? No. In fact, Peter’s quite stable at the very moment. This is the first tip I want to offer you.

The Hungarian Naming Convention

To improve readability in your code, and make it easier for you to look at a project some months after you last worked on it, it would be nice to have some useful names, right? If we would never ever change any names, and just use the ones that VB creates for us, in no time we would get lost. Imagine you have 5 different CommandButtons on your form. 1 for opening files, 1 for saving files, 1 as cancel, 1 for exiting the program, and 1 that makes your screen flash purple. If, in your code, you have to look at Command1 through Command5, you’ll soon be lost. However, if you supplied useful and descriptive names, your code would almost be as easy to read as plain English.

Now, to enhance readability, one of Microsoft’s programmers introduced a naming convention for programming and database objects. The basic idea behind his naming convention is to use a short abbreviation that describes the object before the name of the object as used in your program. For a form, this would be frm, for a CommandButton, it would be cmd. After the prefix, which is written completely in lower case letters, you start the actual name with a capital letter. If you have to use more than one word, you will use capitals to show you started a new word.

Looking back at the earlier example (the 5 buttons), your code would be a lot more readable if you had to look for cmdOpen, cmdSave, cmdCancel, cmdExit, and cmdFlashScreenPurple. In one glance, you would exactly know which button was being used, and it would make the understanding of a particular piece of code a lot better.

If you’re interested in a few articles with lists of widely-used prefixes, check the Microsoft Knowledge base, articles Q173738 and Q110264. This is by no means a list you want to memorize; we will see various prefixes that are commonly used in future articles anyway. One thing to keep in mind though is that you should consistently use the same prefix for the same object throughout all your code.

Back to our calculator. Your Form Designer will now look like this.

We’ll add a TextBox to the form, and 4 buttons. (This will be a very basic calculator that can add, subtract, multiply and divide only).

Set the following properties for the commands.

Leave the caption for all the controls blank. (A quick way to do this is by double clicking on ‘Caption’ in the Properties Window, and then pressing the Delete-key. (Double- clicking will automatically highlight the complete Value)). And for those wondering; Yes, I did do the above scheme in Excel. I suck at HTML-tables :). You’ll be seeing more schemes like that in future articles. Oh, and with ‘Vb-folder, I mean where ever you installed Visual Studio. (Remember, I use Visual Studio Enterprise Edition. You have to have picked ‘Graphics’ during installation. If you do NOT have the graphics, don’t set the picture property, but use the appropriate characters in the Caption property. If you want, change the font size slightly to make the character a little more visible).

Your form ought to look like this by now.

Pretty impressive, huh? Let’s save our work for now. This is where another nifty tip comes along.

Projects folder

I’d strongly recommend making a separate folder where you store your Visual Basic projects. Personally, I use a folder called ‘_Projects’ on a particular drive. The underscore makes sure it’s always the first folder to show up (Unless of course, you’d have a folder with 2 underscores). While that is nice, let’s enhance the tip somewhat. To always start in your newly defined project folder, you need to pull of a crazy little trick. (Otherwise you’ll be doing a lot of clicking; the default start-up folder for Visual Studio is the folder it got installed into. (This tip works on NT or 2000, but I’m quite sure it ought to work on other version of Windows as well).

Find the hyperlink to your VB-icon in the Start-Menu (in your profile). Right-click it.(In the newer Windows versions (NT4, SP 6+, I think), you could also just expand your Program Files folder from the Start Menu, look up VB there, and then right-click the VB-icon). Select Properties. Change the Start-In-folder to your _Projects-folder.

From now on, when you start Visual Basic, it will always offer to save projects in the new folder. I personally prefer to use sub directories for each single project. That just helps to keep things separated and visible.

Back to the calculator. We were going to save our work. Click File / Save As, and save your work somewhere. Notice you’ll be prompted both for the project (which is like the cover of a book; It keeps everything together), and the form (Which could be considered as an element (read: Chapter) of a book).

Well. We have our interface now (An interface is what the user visually gets; In our case, a small form with a text-box and 5 buttons), but we never added any code. If we’d run the code now (use F5, or the button in the ToolBar), we’d see the interface, but it wouldn’t do much. This is where Events step in.

Events

Let’s analyze what a user will be expecting when he sees your interface. He’ll notice he can use the TextBox to enter numbers, and the buttons for performing various mathematical functions. When he types a number made up out of more than one number, nothing has to happen; it’s only when he clicks a button that something will happen. Schematically, the following events will happen.

First, the user will enter a number. Nothing happens, because we cannot be sure of whether or not the user wants to type more numbers; instead, we’ll wait until the user presses a button, and take it that at that point, he wants to perform a calculation on that number. For this week, we will just add functionality to the plus, minus, divide and multiply buttons. We will deal with the ‘=’ button in future articles. To keep things simple, this week we will just make the TextBox show the updated number every time a user clicks one of the buttons.

Now that we know the basic scheme (We’ll enhance this in future articles), let’s look at the events (user-actions) in this scenario. We already determined that when text is entered, nothing will happen. However, when a user clicks a button, we do want something to happen. In Visual Basic terms, this is the definition of an ‘event’. An action, recognized by an object, for which you can write code to respond. . (Taken from the MSDN). For a CommandButton, the click-even is the default event. If you’d double-click the first button (cmdPlus), you’d be taken into the Code Window, and specifically to the ‘cmdPlus_Click()’ event. (Click is the default event for a CommandButton). It means the user supplied a number, and then wants to add another number to it. So what we do is:

# We will take the current value of whatever is in the TextBox.
# We then use add the same number to itself.
# Lastly, we update the TextBox to show the new number.

Not very useful really, but for this example, it will do the trick.

The code for that would be this:

txtInput.Text = txtInput.Text + Val(txtInput.Text)

What we do here is we say the Text-property of the TextBox is equal to the value of the current Text-property, plus the value of the current Text-property. (The period tells Visual basic that we’re referencing an object, and want to access something belonging to that object. (In this case, the Text-property). Note that with value I’m talking about it’s numeric value, and not the value of the Text-property. We use the numeric value, because if a user would input a letter instead of a number, it wouldn’t really make sense. Suppose a user entered the letter ‘a’ instead of a number. a + a would not really make sense to a calculator. Val() is a Visual Basic function designed for that purpose. (We’ll see what Functions are in future articles).

Let me introduce you to another programming concept at this point.

Indentation

If you just typed in the code as supplied, your code would now look like this:

Private Sub cmdPlus_Click()
txtInput.Text = txtInput.Text + Val(txtInput.Text)
End Sub

Imagine hundreds, if not thousands of lines of code in that way. That wouldn’t be readable, would it? Instead, let’s throw in a TAB before the line we typed:

Private Sub cmdPlus_Click()
txtInput.Text = txtInput.Text + Val(txtInput.Text)
End Sub

Notice how that looks much better? It’s like a book; If Tolkien had written lord of the rings in plain text, with no formatting at all, no one would have gotten through the first few pages. Instead, a clever author uses paragraphs, chapters, and what not to make sure that whatever they say is structured. Do the same thing when you’re programming. Make sure that you, or anyone else reading your code will have an easy time reading the code. Take a look at how other people program.

Back to the calculator.

Go back to the Form. (In the Explorer Window, double-click on it). We’ll see the Click- events for our other buttons now. Just go through the same procedure as before (double-click the button, and add the code (Just with different mathematical functions added to it). In the end, your Code-Window will look like this.

Run the program again, punch in some numbers, and use the buttons. You’ll see that by adding code to the ‘Click’ event, we basically told our program what to do whenever a user clicked a button. Other events could be selecting a value from a presented list of options, closing the program, clicking a menu-option, etc. Basically every action a user takes can be seen as an event.

By the way. You might have noticed that when you were typing in the code, after you pressed the dot (‘.’), a little scroll able list popped up.

Intellitype

Because you were assigning a value to the TextBox’s property, Visual Basic recognized you were accessing the TextBox-control, and offered as nice list with possible properties. Quite handy when you’re not actually sure which properties are available, or which one you are looking for. Intellitype also showed up when you were working with the Val-function. It showed you the syntax for the Function there. Intellitype is your best friend while programming; it makes sure you don’t have remember everything about Visual Basic.

ActiveX Data Objects (ADO)

ADO is Microsoft’s latest way of making it easier for us developers to retrieve and work with databases. Don’t worry, you don’t have to be a SQL-guru to use ADO; however, some basics would be nice.

SQL Basics

SQL (Structured Query Language) is a ‘programming’ language used for querying data within Relational Databases. The most basic command is SELECT. SELECT is used to retrieve data from one or more tables of the database. For this article, we will be using the NorthWind database. (It comes with both SQL Server, and MS-Access (For MS-Access, it is installed in the Samples-Folder)).

If we want to retrieve all rows within a certain table, we would issue the following statement:

SELECT * FROM employees

This would retrieve all data from the ’employees’ table. However, if we’d want to see which employee handled which order, we’d query two tables; employees, and orders. How would we know which employee handled which order though? This is where the relational part of the Relational Database-concept kicks in. Instead of listing the employee’s full data (employee-id, name, DOB, etc), the orders-table just shows a reference to a unique number assigned to the employee dealing with a particular order. If we match the number from employees with the number from orders, we would get the full information about the employee, as well as the full order detail. The statement to do this would be:

SELECT * FROM employees, orders WHERE employees.EmployeeID = orders.EmployeeID

Let’s look at that statement for a second. It’s not that hard, and I don’t want to spend too much time on the actual SQL-syntax (Mike handles the SQL Server articles). The first part is similar to our very first statement. SELECT * specifies we want to retrieve all rows from the table(s). FROM employees, orders specifies the tables to retrieve the data from are employees and orders. The comma acts to tell the database we want data from both tables. (If you’d specify more tables, you’d just seperate them with commas each time). WHERE employees.EmployeeID = orders.EmployeeID is what we use to specify the relationship between the tables. In this case the EmployeeID of the orders table is equal to an ID in the employeesID, and we want to see all the information. Does this sound complicated? Let’s look at a quick graphical representation of the tables.

Without going into too much detail, you might see from the diagram (which I whipped up using SQL Server), that between Orders and Employees, a one-to-many reationship exists. One order has one employee handling it. However, an employee can handle many orders. (The way to recognise this is the little symbol looking like two intertwining o’s. this is the mathematical symbol for infinity. (Don’t pin me down on this. Math never was my favourite subject) going to the orders-table).

That ought to be enough about SQL for now. Let’s get started with ADO; after all, that is why you are reading this article.

The ADO Object Model

ADO is a simple way of making a connection with a database, and issuing commands (such as stored procedures in SQL Server), or retrieving data into so-called RecordSets.

ADO resides on top of OLE-DB (from within Visual Basic, at least). OLE-DB enables you to access datasources of different types, without having to deal with issues like network-packets, security, etc.

The ADO Object model graphically looks like this.

Yes, only three objects. Let’s look at them individually, and discuss what they are used for.

Connection Object

The top level object; handles the actual connection to the database. In order for a Connection object to succesfully work, it needs to be told where to connect to, how to connect, and the database to access. This information is stored in it’s ConnectionString property.

Command Object

Using a Command object, you can issue a SQL statement that returns a single row, execute a stored procedure (A prec-compiled SQL statement, or batch of statements on SQL Server), or issue other commands against a database. Commands objects can also support parameters (for instance for useage with a stored procedure). You can use the Command object, without specifying an existing Connection (the Command object will create a Connection object by itself. I strongly recommend not using this approach though, and always creating a Connection object. That way, you can issue multiple commands on the same connection, and you prevent large numbers of connections being created (and hogging up valueable resources on the server)).

RecordSet Object

The RecordSet object defines rows returned from a database; more or less like a database table. A RecordSet is the object we will be using most, if we want to query a database. It is possible to not use an explicit connection when opening a database; however, if you use a connection, you can open multiple RecordSets on the same connection. (Also see above).

A RecordSet is built up out of multiple fields, that each represent a column of data. Graphically, a RecordSet could look like this.

(We created this ‘RecordSet’ by issuing the SQL query SELECT EmployeeID, LastName, FirstName FROM employees). The whole returned list is the RecordSet. The individual ‘boxes’ are all fields. So ‘Anne’ is a field, ‘3’ is a field, and ‘Buchanan’ is a field. (Notice the gray columns are just to provide some readability, and are NOT part of the RecordSet). The total number of fields in this particular RecordSet would be 27. (9 rows of three fields each).

Implementing ADO in an application

Let’s build a quick application to retrieve the RecordSet shown above, shall we?

Start up Visual Basic, and select ‘Standard Exe’ as the project-template.

To display the RecordSet in our program, we’re going to use a Hierarchical Flexgrid. Right-click within the ToolBox, and select Components. Scroll down untill you see ‘Microsoft Hierarchical Flexgrid Component’.

Check the box in front of it to add the component to your Toolbox. In the Toolbox, you’ll notice a new icon:

Draw a new Hflexgrid on your form. The size doesn’t really matter; we can make it fit our RecordSet later.

Now we’re going to add the ADO object-libraries to our project. Click on Project, and select ‘References…’. In the list, scroll down untill you come across ‘Microsoft ActiveX Data Objects 2.1 Library’. (You can also use other versions, instead of 2.1. Just make sure that if you will distribute the application to other users, they also have the same, or a later version).

What exactly did we do here? Well, basically we imported the classes (read: Objects) (The Connection, Command and RecordSet, that is) of ADO into our project, which means as much as that we can now start to use the objects within the ADO-library, along with their events, properties, and functions. (You can get an overview of the complete ADO-library by using the Object Browser. The Object Browser can be called either by using it’s icon in the Toolbar, or by pressing F2.

In the Object Browser, make ADODB the library you want to explore (Set the upper ComboBox’s value to ADODB).

Switch to the code-window (Use either F7, or the ‘View Code’ icon in the Properties Window).

If you followed our Installation Instructions, you ought to see ‘Option Explicit’ there. (If not, Don’t worry. Click on Tools, Options…, and check the ‘Require Variable Declaration’ box). This requires that you declare all variables with the Dim, Public, or Private statement.

In the General Declarations-area (Which is after ‘Option Explicit’ and before any subroutines), we’ll declare two object-variables; 1 for a Connection, and 1 for a RecordSet:

Dim CN As ADODB.Connection
Dim RS As ADODB.RecordSet

Notice that we just allocated memory-space for the variables. The objects themselves do not exist yet. (In other words, the object is set to Nothing). If you don’t like the names I picked, feel free to change them within your own application. CN and RS are just a habit of mine.

Go to the Form_Load() event. (Either double-click on the form itself in the Form Designer, or use the ComboBoxes above the Code-Window. Use the left one to set the current control to ‘Form1’. A form’s default event will be ‘Load’, so you don’t need the right ComboBox to select the event.

Alright. We want to retrieve the employee_id, last name, and first name from the employees table. Remember the SQL statement we used earlier? If not, it’s:

SELECT EmployeeID, LastName, FirstName FROM employees

Our RecordSet will be built with that query. However, we need to supply that query to the RecordSet somehow. In my opinion, the most elegant way to do it is by using a string to store the query in. (You could also supply it directly when opening the RecordSet, but I dislike having really long lines of code. It makes the code less clear to read. That’s a personal decision though. Since I am the one writing this article though, we’ll use my way *sticks out his tongue*).

Dim SQL as String

(Again, SQL is just a habit of mine; I normally prefer using the Hungarian notation (Which I’ll discuss in a beginner-article soon), but for this example, it will do.

Next, we assign the SQL-statement to the string:

SQL = “SELECT EmployeeID, LastName, FirstName FROM employees”

We’re all set to go now. First, we create a connection object:

Set CN = CreateObject(“Adodb.Connection”)

Set means we assign an object reference to a variable (The earlier created CN). CreateObject means we want to create a new instance of the object.

Now we provide the Connection with a ConnectionString, so the connection knows where and how to access the database.

For SQL Server:
CN.ConnectionString = “Provider=SQLOLEDB;Data Source=SQLServer;” & _
“User ID=sa;Password=””;Initial Catalog=NorthWind”

For MS-Access:
CN.ConnectionString=”Provider=Microsoft.Jet.OLEDB.4.0;” & _
“Data Source=C:\Program Files\Microsoft Office\Office\Samples\Northwind.mdb;” & _
“User Id=admin;Password=;”

Note: The ampersand (&) and underscore (_) are used here to increase readability. They allow a string to be spread over multiple lines. You could also just type the full string out on a single line though.

Let’s analyse the connectionstring for a second. Provider supplies the OLE-DB Provider to use. Datasource specifies where ADO should look for the database. In the case of SQL Server, this is where you specify the servername. When using MS-Access, you supply the filepath to the database. User ID is which login to use when you’re accessing the server or database. Since both ‘sa’ in SQL Server and ‘admin’ in MS-Access have a default blank password, there’s no need to specify the Password (meaning you can just leave out the whole Password-string).. If, however, you changed the password, you should provide it