Filter all lines of a text file with multiple strings list

To display only lines containing a string in a text file, easiest way is to use findstr commandline. For example, if we want to display lines containing "ok" (with /i switch for no case sensitivity) in a file file.txt we just to run at command line something like

type file.txt | findstr /i ok

With findstr you can also display lines NOT containing a particular string. For example you want all lines except line with string "not good" with this command

type file.txt | findstr /i /v /c:"not good"

Today my problem is that I need to report all administrators users of all computers except some users in a list.(yes, there is not SCCM or another tool which permit to do it here). I could do it like this:

type file.txt findstr /i /v /c:"administrator1" findstr /i /v /c:"administrator2" findstr /i /v /c:"administrator3"

But that's not really simple to maintain if you need to do some changes. Moreover, why do not do something which could be used for others problems of the same type?
So to do better than excel :) I created this batch file filter.cmd

@echo off
REM
REM Line Filter
REM By F.RICHARD
REM
REM created 2010 July
REM modified 2010 september for %1
REM

SET CURDIR=%~d0%~p0
cd /d %CURDIR%

IF {%1}=={} GOTO :ERRFILEINPUT
SET INPUTFILE=%1
echo Input file:%INPUTFILE%
If NOT EXIST %INPUTFILE% GOTO :ERRNEINPFILE
IF {%2}=={} GOTO :ERRFILEFILTER
SET FILTERFILE=%2
echo Filter file:%FILTERFILE%
If NOT EXIST %FILTERFILE% GOTO :ERRNEFILFILE
IF {%3}=={} GOTO :ERRFILERESULT
SET RESULTFILE=%3
echo Result file:%RESULTFILE%

setlocal EnableDelayedExpansion
set /a nbstr=0

Echo Begin filter
type %INPUTFILE% > tempfile!nbstr!.txt
for /F "tokens=*" %%i IN (%FILTERFILE%) Do (
 set tmpfileread=tempfile!nbstr!.txt
 set /a nbstr=!nbstr!+1 
 set tmpfileresult=tempfile!nbstr!.txt
 type !tmpfileread! | findstr /v /i /c:%%i > !tmpfileresult!
 del /f /q !tmpfileread! 
)
move /Y !tmpfileresult! %RESULTFILE%
Echo End filter
goto :EOF

:ERRFILEINPUT
Echo.
Echo ERROR: you need an input filename
goto :Syntax

:ERRFILEFILTER
Echo.
Echo ERROR: you need a filter file
goto :Syntax

:ERRFILERESULT
Echo.
Echo ERROR: you need a result filename
goto :Syntax

:ERRNEINPFILE
Echo.
Echo ERROR: input file - %INPUTFILE% - does not exist in %CURDIR% directory
goto :EOF

:ERRNEFILFILE
Echo.
Echo ERROR: filter file - %FILTERFILE% - does not exist in %CURDIR% directory
goto :EOF

:Syntax
Echo.
Echo Syntax:
Echo %~nx0 inputfile filterfile resultfile
Echo.
goto :EOF

use it with something like

stringfilter.cmd input.txt filter.txt result.txt

With a input file like this input.txt

aaaaaaa
do not want to see it
bbbbbbb
not this one
ccccccc

This batch use a filter file like this filter.txt

"not this"
bbbbbbb
"not want"

and result file was like this result.txt

aaaaaaa
ccccccc

Et voilà

You can download this script
here

Get all OU's computers in a AD

If you need to have all computers of a specific OU, just run

@dsquery computer "OU=YourOU,DC=Domain,DC=Net" -limit 0 -o rdn > Computers_YourOu_domain_net.txt

Result will be savec in a file Computers_YourOu_domain_net.txt that will be like this:

"computer1"
"computer2"
"computer3"

Now you can use it to run a script on each OU's computers :-)

Append/Consolidate a lot of files to 1 file

Again a need I have regurlarly. You have a lot of files you want to append in only one text. For example a lot of .csv files that you want to consolidate in 1 .csv file

Here it's a powershell script that consolidate all file in a folder to a file "_allFilesInOne.txt". Just put this file GetAllFilesInOne_ps1.txt in a folder and run it (renamed it .ps1 before)

#
# Get All Files In One File
#
# F.Richard
# 2010 June
#
$appendfile = "_allFilesInOne.txt"
$files = Get-ChildItem * -exclude $appendfile,$MyInvocation.MyCommand.Name -include *
$Content = ""
$Content | Out-File $appendfile
ForEach ($filename in $files) {
 Write-Host $filename
 $Content = Get-Content $filename
 $Content | Out-File -append $appendfile
}

You can improve this script easily.
For example, you can exclude .log files by changing
-exclude $appendfile,$MyInvocation.MyCommand.Name
by
-exclude $appendfile,$MyInvocation.MyCommand.Name,*.log

You can can also change inclusion of all files to only .txt files by changing
-include *
by
-include *.txt

Get user SID and vice versa

For a Citrix problem, support team had to know correspondance between user or computer name and SID. If you don't know what is SID, it's something like S-1-5-21-1009609029-4169793705-2371228398 and correspond to an unique identifier of a user, computer or group (see definition in Technet Glossary at http://technet.microsoft.com/en-us/library/cc776189(WS.10).aspx#s)

Well In my toolbox I have 2 very old vbs scripts from Shane Boudreaux that permit to convert SID to username and vice versa

Convert Username to SID
http://www.thescriptlibrary.com/Default.asp?Action=Display&Level=Category3&ScriptLanguage=VBScript&Category1=Security&Category2=Other&Title=Convert%20Username%20to%20SID (Alternate Download)

Convert SID to Username
http://www.thescriptlibrary.com/Default.asp?Action=Display&Level=Category3&ScriptLanguage=VBScript&Category1=Security&Category2=Other&Title=Convert%20SID%20to%20Username (Alternate Download)

But now Microsoft sysinternals have a tool PsGetSid (http://technet.microsoft.com/en-us/sysinternals/bb897417.aspx) (Alternate Download) that do the same

If you need to know how sysinternals do, you can check source code sid2user.cpp and user2sid.cpp from Evgenii B. Rudnyi http://evgenii.rudnyi.ru/soft/sid/ (Alternate Download user2sid.cpp and sid2user.cpp)

If you are a powershell geek, go directly to Powershell Guy blog http://thepowershellguy.com/blogs/posh/archive/2007/01/23/powershell-converting-accountname-to-sid-and-vice-versa.aspx (Alternate Download)

Blog Syntax Highlighting

For my blog, I search something to code Syntax Highlighter (see http://en.wikipedia.org/wiki/Syntax_highlighting if you don't know what that means). Finally, I think there is 2 good choices:

- the simplest:
Use http://www.thecomplex.plus.com/highlighter.html or http://tohtml.com/ to translate your code in your language. Just HTML copy code to you blog .
If you prefer, you can also install a free software: "highlight" from http://www.andre-simon.de/
(Alternate Download)

- the most complete (because it permit to add "copy clipboard" and "view plain" ):
Use http://alexgorbatchev.com/wiki/SyntaxHighlighter (with integration example in http://alexgorbatchev.com/SyntaxHighlighter/integration.html)

return to my blog

I decided to come back to my blog...

How to update or configure automatically vmware esx or linux servers when you are a windows user ?

Problem when you have 32 esx 3.01 to upgrade is that takes long time. And ask to windows prod team to realize these - not interesting - tasks needs a really big documentation to avoid errors. So I decided to create something to automatize this process.
So, today I give you a script which permit you to update or configure any linux/vmware redhat/unix server with simple vbscript file. First, you need at least plink.exe and pscp.exe from http://www.chiark.greenend.org.uk/~sgtatham/putty/ (Alternative Download). Put these files in the same directory than you will put update_esx.vbs file.

But why this script is magic? Because plink and pscp could do it for you really easily, no ? In fact, problem with plink.exe is when you need to automatize your process. Each time you will try to log on a new server, plink will ask you to create ssh host keys. (for security)
With this script you do no need to create plink ssh host keys before running script on your servers. This script do it for you.
Now, you need to create servers.txt file containing all servers you need to update or configure. (1 line = 1 server). For example:

esxserver001
esxserver002

You need also a .sh file which will be executed on esx - linux server (don't forget to use a unix text file with only line feed, and not Windows text file with line feed and carriage return ASCII characters. So do not use notepad to modify files. Use Textpad or other tools that recognize file encoding. )

Below an example to send to your esx. This file ls.sh

#!/bin/sh
# Configure ESX Server
ls

Last thing: root password should be the same on each esx - linux server (else modify my vbscript using a second column in servers.txt like server;rootpassword)
You must also permit root user to log on your esx.

How to use it: run in you windows command line something like

cscript //nologo update_esx.vbs servers.txt file_to_execute.sh your_esx_root_password

If you need to write log, redirect standard output to a file like this:

cscript //nologo update_esx.vbs servers.txt file_to_execute.sh your_esx_root_password > mylog.txt

And now the last but not least, the main tool, update_esx.vbs

'
' update_esx.vbs
'
' by F.RICHARD
'
' v1.00 2007 Nov - Initial release
' v1.01 2008 Jan - Display Standard output
'
Option Explicit

Const ForReading = 1

'
' Detect WScript or CScript
'
If InStr(LCase(WScript.FullName), "wscript")>0 Then  ' ex: FullName = C:\WINNT\system32\wscript.exe 
 WScript.Echo "This script must be run under CScript."
 WScript.Quit
End If

'
' Get Command Line Args
'
Dim Args
Set Args = Wscript.Arguments
If Args.Count < 3  Then
 Wscript.Echo "Syntax : " & WScript.ScriptName & " input_computer_list_file script_to_execute root_password"
 Wscript.Echo "         Example : cscript " & WScript.ScriptName & " servers.txt update.sh password"
 Wscript.Echo ""
 Wscript.Echo "computer_list_file: line begin by # or ; indicate start of a comment, everything after is ignored"
 Wscript.Echo ""
 Wscript.Quit
End If


' Arg(0) = computer list
Dim strInputList
strInputList = Trim(args(0))

' Arg(1) = script to execute
Dim strScriptToExecute
strScriptToExecute = Trim(args(1))


' Arg(2) = root password
Dim strPassword
strPassword = Trim(args(2))


'
' Get Path
'
Dim strScriptFullName, strScriptPath
strScriptFullName = WScript.ScriptFullName ' ex: C:\Program Files\MyFolder\MyProg.exe
strScriptPath = Left(strScriptFullName, InStrRev(strScriptFullName, "\") - 1) ' ex: C:\Program Files\MyFolder

Dim objFso
Set objFso = CreateObject("Scripting.FileSystemObject")


'
' Test If all Files Exist
'
If Not isFileExist(strInputList) Then
 strInputList = strScriptPath & "\" & strInputList
 If Not isFileExist(strInputList) Then
  WScript.Echo "Computer list:" & strInputList & " File Not Exist !"
  WScript.Quit
 End If   
End If   
WScript.Echo "Computer list:" & strInputList & " File Exist"

If Not isFileExist(strScriptToExecute) Then
 strScriptToExecute = strScriptPath & "\" & strScriptToExecute
 If Not isFileExist(strScriptToExecute) Then
  WScript.Echo "Script to execute:" & strScriptToExecute & """" & " File Not Exist !"
  WScript.Quit
 End If   
End If   
WScript.Echo "Script to execute:" & strScriptToExecute & " File Exist"

Dim strPlink
strPlink= strScriptPath & "\" & "plink.exe"
If Not isFileExist(strPlink) Then
 WScript.Echo "PuTTY Link:" & strPlink & " File Not Exist !"
 WScript.Quit
End If     
WScript.Echo "PuTTY Link:" & strPlink & " File Exist"

Dim strPscp
strPscp= strScriptPath & "\" & "pscp.exe"
If Not isFileExist(strPscp) Then
 WScript.Echo "PuTTY Secure Copy client:" & strPscp & " File Not Exist !"
 WScript.Quit
End If     
WScript.Echo "PuTTY Secure Copy client:" & strPscp & " File Exist"


Dim objShell, ComSpec
Set objShell = CreateObject( "WScript.Shell" )
ComSpec=objShell.ExpandEnvironmentStrings("%ComSpec%")

'
' Read computer list file into dictionary
'
Dim objDictionary, return
return = ReadFileToDict(strInputList, objDictionary)
If return = 0 Then
 WScript.Echo "Error during computer list file read"
 WScript.Quit
End If

' Display file
Dim objItem, strServer, objCmd, strCmdline, continue, strGenerateKey
For Each objItem in objDictionary
 strServer = objDictionary.Item(objItem)
 Wscript.Echo ""
 WScript.Echo "Server:" & strServer
 
 ' Test Connection
 strCmdline = ComSpec & " /c """ & strPlink & """ -batch -C -pw " & strPassword & " root" & "@" & strServer & " pwd "
 strGenerateKey = ComSpec & " /c echo y | """ & strPlink & """ " & strServer & " -pw " & strPassword & " root"

 WScript.Echo "Test Connection on server " & strServer
 return = DoConnection(strCmdline, strGenerateKey, strServer, True)

 If (return > 0) Then
  strGenerateKey=""
  Do
   WScript.Echo "Re-verify Connection on server " & strServer
   return = DoConnection(strCmdline, strGenerateKey, strServer, True)
  Loop While (return > 0 and return < 255)
  If (return > 0) Then
   WScript.Echo "Transfer " & strScriptToExecute & " script on server " & strServer
   strCmdline = ComSpec & " /c """ & strPscp & """ -batch -C -pw " & strPassword & " " & strScriptToExecute & " root" & "@" & strServer & ":/tmp"
   return = DoConnection(strCmdline, strGenerateKey, strServer, True)

   WScript.Echo "Execute " & strScriptToExecute & " script on server " & strServer
   strCmdline = ComSpec & " /c """ & strPlink & """ -batch -C -pw " & strPassword & " root" & "@" & strServer & ":/tmp" & " sh /tmp/" & strScriptToExecute
   return = DoConnection(strCmdline, strGenerateKey, strServer, True)
  End If
 End If

Next


'
' Quit
'
WScript.Sleep 1000
WScript.Quit



'--------------------

' DoConnection
'
' 
'
Function DoConnection(strCmdline, strGenerateKey, strServer, displayLogfile)
 Dim objCmd, return, Continue

 'WScript.Echo "cmdline->" & strCmdline
 Set objCmd = objShell.Exec(strCmdline)
 Do While objCmd.Status = 0
  return = objCmd.stdErr.ReadAll
  WScript.Sleep 100
 Loop
 If (displayLogfile) Then
  Dim strStdOut
  strStdOut = objCmd.StdOut.ReadAll()
  If (len(strStdOut) > 0) Then
   Wscript.Echo strStdOut
  End If
 End If

 continue = 0
 If Instr(return, "Access denied" ) > 0 Then ' Access denied
  WScript.Echo "ERROR on server " & strServer & vbCrLf & return
  continue = 0

 Elseif Instr(return, "Unable to open connection" ) > 0 Then ' Unable to open connection
  WScript.Echo "ERROR on server " & strServer & vbCrLf & return
  continue = 0

 Elseif Instr(return, "ERROR" ) > 0 Then ' FATAL ERROR: Network error: Connection timed out
  WScript.Echo "ERROR on server " & strServer & vbCrLf & return
  continue = 0

 Elseif Instr(return, "fingerprint" ) > 0 Then
  If (len(strGenerateKey) > 2) Then
   WScript.Echo "GENERATE key fingerprint on server " & strServer
   'WScript.Echo "generatekey->" & strGenerateKey
   objShell.Exec(strGenerateKey)
   continue = 1
  Else
   WScript.Echo "NOT GENERATE key fingerprint " & strServer
   continue = 2
  End If
 Else
  continue = 255
  'WScript.Echo "Logged on " & strServer & " - " & return
 End if

 DoConnection = continue
End Function



'--------------------

' isFileExist
'
' Test if File Exist
'
Function IsFileExist(strInputFile)
 Dim objFSO
 Set objFSO = CreateObject("Scripting.FileSystemObject")
 If Not objFSO.FileExists(strInputFile) Then
  Wscript.echo strInputFile & " file not exist !"
  IsFileExist = False
 Else
  IsFileExist = True
 End If
End Function


'--------------------

' ReadFileToDict
'
' Read each line -> put in Dictionary
' strInputfile = file to read
' objDictionary = dictionary for results
'
Function ReadFileToDict(strInputFile, ByRef objDictionary)
 Dim objFSO, objTextFile, strNextLine, i
 Set objDictionary = CreateObject("Scripting.Dictionary")
 Set objFSO = CreateObject("Scripting.FileSystemObject")
 If isFileExist(strInputFile) Then
  Set objTextFile = objFSO.OpenTextFile(strInputFile, ForReading)
  i = 0
  Do Until objTextFile.AtEndOfStream
   strNextLine = Trim(objTextFile.Readline)
   If ( (Not Left(strNextLine, 1) = "#") _
    and (Not Left(strNextLine, 1) = ";") _
    and (Len(strNextLine)>1) _
    ) Then
    objDictionary.Add i, strNextLine
    i = i + 1
   End If
  Loop
  objTextFile.Close
  ReadFileToDict = True
 Else
  ReadFileToDict = False
 End If
End Function


'--------------------

You can download a .zip file here including an example with change_ntp.sh

Next step, if I had some time, a graphical interface with a description of each script and variables for each script.