Contacts import using runspacepools - multithreading

Question

I'm trying to import contacts using RunspacePools, but I'm having trouble getting it to work. If I take it out of the runspace logic, it works fine, just takes a long time. I'd really like to use runspacepools to speed up the import process and make it run multithreaded so it imports faster. On avg each import takes about 5-6 mins per user, and I have about 500 users, so it can take up to 3000 mins to run.
Here is what I currently have:
#---------------------------------------------
. $rootPath\UpdateContacts\UpdateContacts.ps1
# Set up runspace pool
$RunspacePool = [runspacefactory]::CreateRunspacePool(1,10)
$RunspacePool.Open()
# Assign new jobs/runspaces to a variable
$Runspaces = foreach ($User in $Users)
{
# Create new PowerShell instance to hold the code to execute, add arguments
$PSInstance = [powershell]::Create().AddScript({
$Users | ForEach{ UpdateContacts($_) }
}).AddParameter('$_')
# Assing PowerShell instance to RunspacePool
$PSInstance.RunspacePool = $RunspacePool
# Start executing asynchronously, keep instance + IAsyncResult objects
New-Object psobject -Property #{
Instance = $PSInstance
IAResult = $PSInstance.BeginInvoke()
Argument = $User
}
}
# Wait for the the runspace jobs to complete
while($Runspaces |Where-Object{-not $_.IAResult.IsCompleted})
{
Start-Sleep -Milliseconds 500
}
# Collect the results
$Results = $Runspaces |ForEach-Object {
$Output = $_.Instance.EndInvoke($_.IAResult)
New-Object psobject -Property #{
User = $User
}
}
And my "UpdateContacts.ps1" file looks like this:
Function UpdateContacts($User)
{
Write-host "Importing Contacts. This could take several minutes."
#FirstName, MiddleName, LastName, DisplayName, SamAccountName, Email, Mobile, TelephoneNumber, Title, Dept, Company, Photo, ExtensionAttribute2
$ContactsBody = #"
{
"givenName" : "$($User.FirstName)",
"middleName" : "$($User.MiddleName)",
"surname" : "$($User.LastName)",
"displayName" : "$($User.DisplayName)",
"jobTitle" : "$($User.Title)",
"companyName" : "$($User.Company)",
"department" : "$($User.Dept)",
"mobilePhone" : "$($User.Mobile)",
"homePhones" : ["$($User.TelephoneNumber)"],
"emailAddresses":
[
{
"address": "$($User.Email)",
"name": "$($User.DisplayName)"
}
]
}
"#
Try
{
Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId/contacts" -Headers $headers -Body $ContactsBody -Method Post -ContentType 'application/json' | Out-Null
#After each user clear the info
$User = $NULL
}
Catch
{
if($error)
{
$User
$error
pause
}
$_.Exception.Message
Write-Host "--------------------------------------------------------------------------------------"
$_.Exception.ItemName
}
}
Any help is appreciated.
EDIT:
Here is the full script (with the exception of the ContactUploader.ps1 script. That function is in a separate script but the whole code (Function) is posted above).
CLS
##################### Import Thread/Job Module to perform multithreading #####################
if(!(Get-Module -ListAvailable -Name ThreadJob))
{
$NULL = Install-Module -Name ThreadJob -Scope CurrentUser -Force -Confirm:$False
}
##################### ------------------------------------------------------------- #####################
#Root Path
$rootPath = $(Split-path $MyInvocation.MyCommand.path -Parent)
#Prevent connection from closing on us when we use "Invoke-RestMethod"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12
Add-Content "$rootPath\progress.txt" ""
Add-Content "$rootPath\progress.txt" "********** Starting Script $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
##################### Connect to Microsoft Graph API and configure all of our Variables #####################
Add-Content "$rootPath\progress.txt" "********** Connecting to Graph API $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$ApplicationID = "ApplicationID"
$TenatDomainName = "domain.com"
$AccessSecret = "ItsASecret"
$global:Body = #{
Grant_Type = "client_credentials"
Scope = "https://graph.microsoft.com/.default"
client_Id = $ApplicationID
Client_Secret = $AccessSecret
}
$ConnectGraph = Invoke-RestMethod -Uri "https://login.microsoftonline.com/$TenatDomainName/oauth2/v2.0/token" -Method POST -Body $Body
Add-Content "$rootPath\progress.txt" "********** Finished Connecting to Graph API $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$global:token = $ConnectGraph.access_token
$global:UPN = "user#domain.com"
$global:AccessToken = $token
$global:User = $NULL
$global:Contact = $NULL
$global:NeedsToBeAdded = $NULL
$global:NeedsToBeDeleted = $NULL
$global:folderId = $NULL
$global:NewContactFolder = $NULL
$global:FolderName = "Test Contacts"
$global:headers = #{
"Authorization" = "Bearer $AccessToken"
"Accept" = "application/json;odata.metadata=none"
"Content-Type" = "application/json; charset=utf-8"
"ConsistencyLevel" = "eventual"
}
#Create Contact Folder if it doesn't exist
$global:ContactsFolderBody = #"
{
"parentFolderId": "$ParentFolderID",
"displayName": "Test Contacts"
}
"#
Add-Content "$rootPath\progress.txt" "********** Grabbing Contact Folder Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$global:folders = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders" -Headers $headers
$global:ParentFolderID = $folders[0].value.parentFolderId
#Get Folder ID we are working with
foreach($folder in $folders.value)
{
#Reset the Value
$folderId = $NULL
if($FolderName -eq $folder.displayName)
{
$folderId = $folder.id
break
}
}
Add-Content "$rootPath\progress.txt" "********** Finished Grabbing Contact Folder Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
##################### Check if our Contacts Folder exists. If it doesn't, create it. #####################
if($NULL -eq $folderId)
{
Add-Content "$rootPath\progress.txt" "********** Creating Contact Folder $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$Start = Get-Date
Write-Host "Creating Contacts Folder"
Try
{
while($NULL = (Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId" -Headers $headers -Method get))
{
$NewContactFolder = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders" -Body $ContactsFolderBody -Headers $headers -Method post -ContentType 'application/json'
sleep -Milliseconds 1
$folderId = $($NewContactFolder.id)
}
}
Catch
{
Out-Null
}
$End = Get-Date
Write-Host "Contacts Folder created in $($Start - $End) seconds"
Write-Host ""
Add-Content "$rootPath\progress.txt" "********** Finished Creating Contact Folder $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
}
##################### Grab all of our User Information from AD #####################
Add-Content "$rootPath\progress.txt" "********** Grabbing AD User Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$Start = Get-Date
$searcher=[adsisearcher]""
$searcher.Sort.PropertyName = "sn"
$searcher.Filter = "(&(objectcategory=person)(objectclass=user)(extensionAttribute2=custom)(|(mobile=*)(telephonenumber=*)))"
$colProplist = #(
'givenname', 'extensionattribute2'
'initials', 'mobile', 'telephonenumber'
'sn', 'displayname', 'company'
'title', 'mail', 'department'
'thumbnailphoto', 'samaccountname'
)
$colPropList | & { process {
$NULL = $searcher.PropertiesToLoad.Add($_)
}}
$End = Get-Date
Write-Host "User info took $($Start - $End) seconds"
Write-Host ""
Add-Content "$rootPath\progress.txt" "********** Finished Grabbing AD User Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
##################### Create our User Hashtable #####################
Add-Content "$rootPath\progress.txt" "********** Creating User Hashtable $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
Write-Host "Creating User Hashtable"
$Start = Get-Date
$users = $searcher.FindAll() | & { process {
[pscustomobject]#{
FirstName = [string]$_.properties.givenname
MiddleName = [string]$_.properties.initials
LastName = [string]$_.properties.sn
DisplayName = [string]$_.properties.displayname
SamAccountName = [string]$_.properties.samaccountname
Email = [string]$_.properties.mail
Mobile = [string]$_.properties.mobile
TelephoneNumber = [string]$_.properties.telephonenumber
Title = [string]$_.properties.title
Dept = [string]$_.properties.department
Company = [string]$_.properties.company
Photo = [string]$_.properties.thumbnailphoto
ExtensionAttribute2 = [string]$_.properties.extensionattribute2
}
}}
Write-Host "User Hashtable took $($Start - $End) seconds"
Write-Host ""
Add-Content "$rootPath\progress.txt" "********** Finished Creating User Hashtable $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
##################### Get Existing Contacts (Only if the Contacts Folder wasn't newly created )#####################
if($NULL -ne $folderId)
{
Add-Content "$rootPath\progress.txt" "********** Grabbing Contact Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$Start = Get-Date
$AllContacts = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId/contacts?`$top=999&`$Orderby=Surname" -Headers $headers -Method Get
$End = Get-Date
Write-Host "Contact info took $($Start - $End) seconds"
Write-Host ""
Add-Content "$rootPath\progress.txt" "********** Finished Grabbing Contact Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
##################### Create our Contact Hashtable #####################
Add-Content "$rootPath\progress.txt" "********** Creating Contact Hashtable $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
Write-Host "Creating Contact Hashtable"
$Start = Get-Date
$Contacts = $AllContacts.value | & { process {
[PSCustomObject]#{
'FirstName' = [string]$_.givenName
'MiddleName' = [string]$_.initials
'LastName' = [string]$_.surname
'DisplayName' = [string]$_.displayName
'Email' = [string](($_.emailAddresses) | %{$_.Address})
'Mobile' = [string]$_.mobilePhone
'TelephoneNumber' = [string]$_.homePhones
'Title' = [string]$_.jobTitle
'Dept' = [string]$_.department
'Company' = [string]$_.companyName
}
}}
$End = Get-Date
Write-Host "Contact HashTable took $($Start - $End) seconds"
Write-Host ""
Add-Content "$rootPath\progress.txt" "********** Finished Creating Contact Hashtable $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
}
##################### Start Comparing Data #####################
#Our Array of values we will be comparing
[array]$CompareValues = "FirstName","MiddleName","LastName","DisplayName","Email","Mobile","TelephoneNumber","Title","Dept","Company"
for($i=0; $i -lt $CompareValues.Count; $i++)
{
#First let's create 2 variables that will hold the info we want
$A = ($Users).($CompareValues[$i])
$B = ($Contacts).($CompareValues[$i])
##################### Update Contacts #####################
#Only Run if there are contacts; otherwise there is nothing for us to compare
if(($NULL -ne $B))
{
#Displays all differences
#$Differences = [string[]]([Linq.Enumerable]::Except([object[]]$a, [object[]]$b) + [Linq.Enumerable]::Except([object[]]$b, [object[]]$a))
#Displays what accounts we need to import
$NeedsToBeAdded = [string[]]([Linq.Enumerable]::Except([object[]]$a, [object[]]$b))
#Displays what accounts we need to delete because they no longer exist
$NeedsToBeDeleted = [string[]]([Linq.Enumerable]::Except([object[]]$b, [object[]]$a))
}
##################### Import All Contacts #####################
Else
{
Add-Content "$rootPath\progress.txt" "********** Importing All Contacts $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
$Start = Get-Date
<#
#---------------------------------------------
. $rootPath\UpdateContacts\UpdateContacts.ps1
# Set up runspace pool
$RunspacePool = [runspacefactory]::CreateRunspacePool(1,10)
$RunspacePool.Open()
# Assign new jobs/runspaces to a variable
$Runspaces = foreach ($User in $Users)
{
# Create new PowerShell instance to hold the code to execute, add arguments
$PSInstance = [powershell]::Create().AddScript({
$Users | ForEach{ UpdateContact($_) }
}).AddParameter('$_')
# Assing PowerShell instance to RunspacePool
$PSInstance.RunspacePool = $RunspacePool
# Start executing asynchronously, keep instance + IAsyncResult objects
New-Object psobject -Property #{
Instance = $PSInstance
IAResult = $PSInstance.BeginInvoke()
Argument = $User
}
}
# Wait for the the runspace jobs to complete
while($Runspaces |Where-Object{-not $_.IAResult.IsCompleted})
{
Start-Sleep -Milliseconds 500
}
# Collect the results
$Results = $Runspaces |ForEach-Object {
$Output = $_.Instance.EndInvoke($_.IAResult)
New-Object psobject -Property #{
User = $User
}
}
#---------------------------------------------
#>
Write-host "Importing Contacts. This could take several minutes."
#There are no contacts, so let's import them
#Path to our script that imports Contacts
. $rootPath\UpdateContacts\UpdateContacts.ps1
#$Users | & { process { UpdateContacts($_) } }
#Start-ThreadJob -ScriptBlock { $Users | & { process { UpdateContacts($_) } } }
Start-ThreadJob -ScriptBlock { $Users | ForEach{ UpdateContacts($_) } }
Get-Job
$End = Get-Date
Write-Host "Contact Import took $($Start - $End) seconds"
Write-Host ""
Add-Content "$rootPath\progress.txt" "********** Finished Importing All Contacts $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""
break
}
}
Add-Content "$rootPath\progress.txt" "********** Finished Script $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

Answer 1

There is a bunch of code to go through so I'm gonna give you a blueprint of how you can achieve processing all users in $users using ThreadJob.
So, step by step, I'll try to add as much comments as I consider appropriate to guide you through the thought process.
I'm not sure what is the output of your function since I see an | Out-Null at the end of the Invoke-RestMethod. You would need to clarify on this.
# requires -Modules ThreadJob
# Load UpdateContacts function in memory
. "$rootPath\UpdateContacts\UpdateContacts.ps1"
# Save the function in a scriptBlock, we need this
# so we can pass this function in the scope of the ThreadJobs
$updateContacts = "function UpdateContacts { $function:updateContacts }"
# Define the Number of Threads we are going to use
# (Get-CimInstance win32_processor).NumberOfLogicalProcessors
# Can give you a good perspective as to how many Threads is safe to use.
$numberOfThreads = 10
# I'm assuming that $users is the array we want to process with
# the UpdateContacts function. Around 500 as you said in your question.
# Here I'm grouping the users in chunks so each running Job can process
# a chunk of users. Each chunk will contain around 50 users to process.
$groupSize = [math]::Ceiling($users.Count / $numberOfThreads)
$counter = [pscustomobject]#{ Value = 0 }
$chunks = $users | Group-Object -Property {
[math]::Floor($counter.Value++ / $groupSize)
}
# Here is the magic
foreach($chunk in $chunks)
{
# Capture this chunk of users in a variable
$thisGroup = $chunk.Group
# This is what we are running inside the scope
# of our threadJob
$scriptBlock = {
# As in my comments, these variables don't exist inside here,
# you need to pass them to these scope
$UPN = $using:UPN
$folderID = $using:folderId
$headers = $using:headers
$contactsBody = $using:contactsBody
# First we need to define the function inside
# this scope
. ([scriptBlock]::Create($using:updateContacts))
# Loop through each user
foreach($user in $using:thisGroup)
{
UpdateContacts -User $user
}
} # EOF Job's ScriptBlock
# ThrottleLimit is the number of Jobs that can run at the same time.
# Be aware, a higher number of Jobs running does NOT mean that the
# task will perform faster. This always depends on your CPU & Memory.
# And, this case in particular, the number of requests your URI is able to handle
Start-ThreadJob -ScriptBlock $scriptBlock -ThrottleLimit $numberOfThreads
}
# Now we should have 10 Jobs running at the same time, each Job
# is processing a chunk of 50 users aprox. (500 users / 10)
# Note: As in my previous comments, I see an Out-Null in your function
# not sure what is meant to return but in case, this is how you capture
# the output of all Jobs:
$result = Get-Job | Receive-Job -Wait
# Free up memory:
Get-Job | Remove-Job