Microsoft Dynamics CRM 2016 On-Premise Deployment on Azure VM – Part 2 – Internet-Facing Deployment Configuration

Objectives

This article in 2 parts, explains step by step how to set up a Microsoft Dynamics CRM 2016 development server on an Azure Virtual Machine and to open it to the Internet via the CRM Internet-Facing Deployment.

These steps are not intended to be followed for a Production server as they rely on a single Virtual Machine for all the infra-structure including:

  • Windows Server 2012 R2 Operating System
  • Active Directory Domain Services role,
  • Web Server (IIS) Role
  • Active Directory Federation Service (ADFS) V3.0 role,
  • SQL Server Enterprise 2014 SP1,
  • and of course Dynamics CRM 2016 with all roles on the same server.

This is clearly not a configuration recommended and supported by Microsoft but it’s very didactic as it touches to a good range of Microsoft server technologies and concepts.

This deployment adds a bit of complexity regarding some required work-arounds to make it run properly, but also remove some complexity in overall (less servers to deploy and configure, lower administration). Some straight lines and assumptions are taken, for example regarding the Windows accounts that again is not suitable for a Production environment.

Part1 covers the core infrastructure installation with MS Dynamics CRM 2016 configured for a simple AD Authentication, restricted to internal users. The schema below highlights the process:

You can complete Part 1 of this article and stop there with a fully functional Dynamics CRM 2016 server accessible only from the internal network.

Part 2 covers the Internet-Facing Deployment configuration of Dynamics CRM 2016 to have it support Claim-Based authentication for internal and external access thanks to the MS Active Directory Federation Service. The schema below highlights the process:

Requirements

For this deployment you need:

  • A SSL wildcard certificate provided by a well know Certificate Authority,
  • Admin access to a domain provided by an Internet Domain Name Registration Provider or by your company (access to the DNS),
  • Full Administration access on the infrastructure of your servers,
  • If you are building a development environment hosted on Azure, then the Part 1 of this article is assumed to be covered.

SSL Certificate Application

Our development instance of Dynamics CRM will be accessed via the URL https://crm2016.mydomain.com:444 (In this article mydomain.com is a fake domain to be replaced by your domain of choice).

We need to purchase a standard SSL wildcard certificate to be applied to our IIS Server. If you intend to stop this deployment to Dynamics CRM only, then you could eventually consider to use a free self-signed certificate for testing/dev purpose, but be aware that some integrations with other systems, following this article, may not work.

As a SSL Certificate Provider, we’ll use the services of https://www.namecheap.com that is one of the cheaper provider of the market with pretty good tracking records.

On your Windows Server, open IIS Manager:

  • Click the Server Name of the left panel
  • On the central panel, double click the ‘Server Certificates’ icon
  • On the right panel, click ‘Create Certificate Request’

Fill-up the ‘Distinguished Name Properties’ pop-up fields with alphanumeric symbols (no special ones such as ‘&’, ‘/’,’@’, etc. allowed):

  • Common Name – The name through which the certificate will be accessed (example *.mydomain.com for a wildcard certificate)
  • Organization – The legally registered name of your organization/company
  • Organizational unit – The name of your department within the organization
  • City/locality – The city in which your organization is located
  • State/province – The state in which your organization is located
  • Country/region – two-digit country code

The next screen ‘Cryptographic Service Provider Properties’ offers to choose 2 parameters. Check with your Certificate Provider in order to be sure what values to fill there. The most common values today are:

  • Provider = ‘Microsoft RSA SChannel Cryptographic Provider’
  • Bit length = 2048

Next select a place where to store the Certificate Request text file, any place and any file will work.

Now on the SSL Certificate Provider web site, activate your newly purchased certificate and paste the content from the previously generated file to the Certificate Signing Request (CSR) field. Select IIS as a web server if this choice is available.

Your SSL Certificate provider will require to go through a Domain Control Validation (DCV) process for security purpose.

We opted for the Email method to complete the DCV (Other possible alternative: HTTP-based validation or DNS-based validation). The validation can take a few hours to a couple of days.

Once the SSL Certificate delivered, download it and select the Action “Complete Certificate Request” in IIS Manager and upload your certificate.

Our SSL Certificate Provider delivered the certificate under different formats, we selected the one with the extension p7b (.cer, .p7s, .p7b should be equally accepted by IIS).

Web Sites Binding with SSL Certificate

Default Site

On your Windows Server, open IIS Manager:

  • Click the Server Name of the left panel
  • Expand the Sites and select the Default Web Site
  • On the right Action panel, click “Bindings”
  • Click the button “Add” in the Site Bindings” pop-up

  • Select Type = ‘https’
  • Leave IP address to ‘All Unassigned’
  • Set Port = 443
  • Select your wildcard SSL Certificate by its friendly name
  • Click OK

Dynamics CRM Site

in IIS Manager:

  • Click the Server Name of the left panel
  • Expand the Sites and select the Microsoft Dynamics CRM Web Site
  • On the right Action panel, click “Bindings”
  • Click the button “Add” in the Site Bindings” pop-up

  • Select Type = ‘https’
  • Leave IP address to ‘All Unassigned’
  • Set Port = 444 (must be different from the port used for the default web site)
  • Select your wildcard SSL Certificate by its friendly name
  • Click OK

DNS Configuration

Our DNS requirements are the following:

  • To set up Claims-based authentication for our Dynamics CRM server:
    • internalcrm.mydomain.com – URL accessed by internal network users à Internal access
    • sts.mydomain.com – URL pointing to the security token service (ADFS) à Internal and external access
  • To set up IFD:
    • crm2016.mydomain.com – URL accessed by external network users à External access

      Important: this name has to match the name of your Dynamics CRM Organization

    • dev.mydomain.com – Discovery service URL à External access
    • auth.mydomain.com – external IFD URL à External access

We need the services of an Internet Domain Name Registration Provider to create an external domain. Typically, providers such as Namecheap (http://www.namecheap.com) or GoDaddy (http://godaddy.com) can provide this for a reasonable cost.

For External accesses, we create the following A Records pointing to our public IP (See Part 1 of this article to find out how did we get this IP from Azure) From the user interface of the Domain Provider.

For internal accesses, we can set up A Records via our local DNS Manager…

… or add the following entries within the host.file. This file is located on your loca server: C:\Windows\System32\drivers\etc. This is acceptable for a development server.

(replace holes by your domain and 10.0.0.4 by your private IP adress)

Test all the DNS names, from your server and from the internet.

Notes:

  • You may not be able to ping those names because Azure may block those request
  • In case of local name resolution problems, you can try to clean up the DNS cache

Open your Server to the Internet

The following steps are specific to the Dynamics CRM server deployed on an Azure VM (Part 1 of this article). If your server is hosted by a different Cloud platform provider, then you’ll need to refer to their documentation. In any cases, the second step about configuring the Firewall stay valid.

Inbound Security Rules on Azure

Create Inbound Security Rules on your Network Security Group to allow inbound access from the outside world.

On the Azure Portal:

  • Select your Virtual Machine
  • Select your current Network Interface associated with your Static IP
  • Select the Network Security Group
  • Click on the “Inbound Security Rules”
  • Create the following rules depending on your need
    • Open Port 80 (only if access to default web site on HTTP is required)
      • Source:any / protocol:TCP / Source port range:* / Destination:any / Destination port range: 80 / Action: Allow
    • Open Port 443 (required for our Dynamics CRM IFD set up where port 443 is bound to the default web site for ADFS)
      • Source:any / protocol:TCP / Source port range:* / Destination:any / Destination port range: 443 / Action: Allow
    • Open Port 444 (required for our Dynamics CRM IFD set up where port 444 is bound to the Dynamics CRM web site)
      • Source:any / protocol:TCP / Source port range:* / Destination:any / Destination port range: 444 / Action: Allow

Firewall configuration

On the Windows Server, launch the “Windows Firewall with Advanced Security” Tool from the Server Manager and create the following Inbound Rules reflecting the rules you have defined via your Azure Network – Security Group:

  • Open port 443 on TCP
  • Open port 444 on TCP
  • Optionally if you want to leave access to your default web site in HTTP, open port 80 on TCP

The firewall could eventually cause problems for the remaining steps of this deployment if not configured properly, we recommend to temporarily turn it off until you can verify the IFD is working properly with Dynamics CRM and then turn it back on again later.

Declare your Domain’s Web sites in Internet Explorer

Include all your web sites in the local intranet zone in order to avoid some security blockages with Internet Explorer later.

Within IE, click on the “Tools” Icon and select “Internet Options”.

Pick the tab “Security”, select “Local Intranet” and click the “Sites” button, and then click the “Advanced” button on the pop-up.

Finally, add all the web sites of your domain, e.g. add the entry “*.mydomain.com”.

Claim-Based Authentication Configuration for Internal Access

Active Directory Federation Services Installation and Configuration

We need to install ADFS 3.0 that will act as the Security Token Service (STS) for our Claim-based authentication. It will be using our default IIS web site.

On your Server, open the Server Manager:

  • Click “Add Roles and Features”
  • In the wizard, select “Server Roles”
  • Check the role “Active Directory Federation Services”

  • In the wizard, select “ADFS” and read the content
  • In the wizard, select “Confirmation” and tick the box “Restart the destination server automatically if required”
  • Click “Install”

Once the installation is complete, click “Configure the federation service on this server”.

On the Configuration wizard, leave the choice “Create the first federation server farm” and click “Next”.

Next screen, select an account with AD domain admin permissions, your account should be fine in the context of this development installation.

On the “Specify Service Properties” screen:

  • Select the SSL certificate we created at the beginning of this article (do not click Import)
  • Set the Federation Service Name (e.g. sts.mydomain.com)
  • Choose a relevant FS Display Name as the users will see it at sign in (something like “MS Dynamics CRM”

If you click Next, you’ll see the following error:

Open Windows PowerShell and execute the following command:

  • Add-KdsRootKey -EffectiveTime(Get-Date).AddHours(-10)

Now you can click “Previous” and then click “Next” on the wizard if you see a Guid provided as an answer.

You can go for the creation of a Group Managed Service Account since you are on a development environment. It would be recommended to use a defined account in Production.

On the next screen we create a database on the server using Windows Internal Database, but you can choose to leverage the existing SQL Server on your Windows Server.

Click “Next” and review your selections and then go to the Pre-requisite Checks.

You can safely ignore the 1st warning, as long as All prerequisite checks passed successfully. Click “Configure”.

Finally, if everything goes well, you should obtain the success confirmation screen.

Let’s check the URL of the ADFS metadata in Internet Explorer to verify the ADFS is working properly; you’ll need this URL later.

It should look like the URL below:

https://sts.mydomain.com/federationmetadata/2007-06/federationmetadata.xml

  • Replace “sts” eventually by another DNS name of your choice dedicated to ADFS (done in step “DNS Configuration”)
  • Replace “mydomain.com” by your own domain
  • If Internet Explorer doesn’t display the XML file, make sure you have done the steps described in section “Declare your Domain’s Web sites in Internet Explorer”
  • Another important thing to check is that no error appears on the browser regarding your SSL certificate (click on the locker icon close to the URL for more info about it)

Configure MS Dynamics CRM Server for Claims-Based Authentication

Open the CRM Deployment Manager and follow these steps:

  • Select “Properties” in the Action menu
  • Go to the “Web Address” tab

  • Change Binding Type from HTTP to HTTPS
  • Replace all occurrence of the server name by the URL accessed by internal network users internalcrm.mydomain.com (see step “DNS Configuration”)
  • Change all occurrences of the port 5555 to 444

  • Click the “Apply” button and close the pop-up
  • Back to the Microsoft Dynamics CRM Deployment Manager, right click on the top of the tree “Microsoft Dynamics CRM” and select in the menu “Configure Claims-Based Authentication”

  • Click Next on the welcome page of the wizard
  • On the “Specify the security token service”, enter the Federation metadata URL we tested in the previous steps, setting up ADFS

  • Click “Next” and pick your SSL Certificate declared previously with a friendly name

  • Click “Next” and review the System Checks, you should have 2 successes à this stage

  • Click “Next” and then click “Apply”
  • On the last screen, click “View the log file”

  • Click “Finish” on the wizard
  • Test that the Internal Federation Metadata URL is working (You may have to do the next step prior to test successfully this URL)

Next we need to grant to the account NETWORK SERVICE the access to the encryption certificate because it’s the account that has been associated by default to the CRMAppPool in IIS. You can double check it on the Application Pools in IIS.

  • From the Windows Start button, right click and select “Run”
  • Type the command “mmc” to launch the Microsoft Management Console

  • Select “Add/Remove Snap-in” from the “File” menu

  • Select the “Certificates” snap-in and click “Add”

  • Select “Computer Account” and click “Next”
  • Make sure “Local computer” is selected and click “Finish” and “OK”

  • Select “Console Root” à “Certificates” à “Personal” à “Certificates”
  • Right click on your SSL Certificate we declared for the CRM and select “All Tasks” à “Manage Private Keys”

  • Click the “Add” button
  • Click the “Advanced” button
  • Type a search string like “network service” and click “Find Now” button
  • Pick the account “NETWORK SERVICE” and click “OK”

  • Ensure the NETWORK SERVICE has got Read access

  • Apply and close MMC.

Create and configure Claims Provider Trusts and Relying Party Trusts in ADFS

Start AD FS Management from the Server Management’s Tools and follow these steps:

  • Select “ADFS” à “Trust Relationships” à “Claims Provider Trusts” on the side panel
  • Right click on “Active Directory” and select “Edit Claim Rules…” on the menu

  • In the Claim Rules editor click the button “Add Rule”

  • Select “Send LDAP Attributes as Claims” for the Claim rule template and click “Next”

  • In the Claim Rule configuration screen, create the following rule:
    • Claim rule name: UPN Claim Rule
    • Attribute store: Active Directory
    • LDAP Attribute: User Principal Name
    • Outgoing Claim Type: UPN

  • Click “Finish” and “OK” to leave the Claim Rules editor

Now it is time to configure the CRM as a relying party to consume claims from ADFS.

Stay in the ADFS Management and follow these steps:

  • Select “ADFS” à “Trust Relationships” à “Relying Party Trusts” on the side panel
  • Click “Add Relying Party Trust” in the right Actions panel

  • Click “Next” and specify a display name in the following screen like “CRM Claims Relying Party”

  • Click “Next” and do not configure the “Configure Multi-factor Authentication”, so click “Next” again
  • On the step “Choose Issuance Authorization Rules”, leave the option “Permit all users to access this relying party” checked

  • Click “Next” and leave the step “Ready to Add Trust” as it is by clicking again “Next”

  • On the “Finish” step, make sure to check the tick box “Open the Edit Claim Rules dialog for this relying party trust when the wizard closes” and push the “Close” button

  • On the Claim Rule Editor, click the button “Add Rule”
  • Select the Claim rule template “Pass Through or Filter an Incoming Claim” and click “Next”

  • Create the Rule n°1:
    • Claim rule name: Pass Through UPN
    • Incoming claim type: UPN
    • Check “Pass through all claim values”
  • Click “Finish”

  • On the Claim Rule Editor, click the button “Add Rule” again
  • Select the Claim rule template “Pass Through or Filter an Incoming Claim” and click “Next”
  • Create the Rule n°2:
    • Claim rule name: Pass Through Primary SID
    • Incoming claim type: Primary SID
    • Check “Pass through all claim values”
  • Click “Finish”
  • On the Claim Rule Editor, click the button “Add Rule” again
  • Select the Claim rule template “Transform an Incoming Claim” and click “Next”
  • Create the Rule n°3:
    • Claim rule name: Transform Windows Account Name to Name
    • Incoming claim type: Windows account name
    • Outgoing claim type: Name
    • Check “Pass through all claim values”

  • Click “Finish”
  • Back to the Claim Rules editor, check you have your 3 rules and click “Apply” and “OK”

We need now to enable Forms authentication for internal access which is off by default.

Stay in the ADFS Management and follow these steps:

  • Select “ADFS” à “Authentication Policies” on the side panel
  • Click “Edit” on section “Primary Authentication” – “Global Settings” – “Authentication Methods”

Check the box “Forms Authentication” in the Intranet section.

Apply and close the ADFS Management console.

Open Internet Explorer and open the Internet Options.

Open the tab “Advanced” and make sure the option “Enable Integrated Windows Authentication” is checked.

This should be done for every PC accessing the internal access points so that ADFS and CRM can pass the Kerberos tickets without being prompted for credentials.

Finally, we need to register the ADFS Service Principal Names (SPN) on the user running the ADFS Service:

  • Open PowerShell or CMD
  • Enter the following command:
  • setspn -a HTTP/sts.mydomain.com mydomain\Administrator
  • setspn -a HOST/sts.mydomain.com mydomain\Administrator

(adjust those parameters with your domain and the user you choose to run the ADFS Service)


  • Reset IIS with the command
  • iisreset

Before to move to the IFD configuration, you can test the internal access to Dynamics CRM by typing the URL https://internalcrm.myserver.com:444.

If it doesn’t work at this stage, you need to troubleshoot it before to move on to the next stage. The SPN config. is critical, make sure it’s done correctly.


Internet-Facing Deployment Configuration for External Access

Configure MS Dynamics CRM Server for Internet-Facing Deployment

Open the Dynamics CRM Deployment Manager:

  • Right Click on “Microsoft Dynamics CRM” and pick “Configure Internet-Facing Deployment” in the contextual menu

  • Click “Next” on the 1st screen
  • On the next screen, enter the URLs with the port 444 for
    • the Web Application server domain (it’s a domain, not a server)
    • the Organization Web Service domain (identical to web application since we are installing all CRM roles on the same server)
    • The Discovery Web Service Domain (here we need a resolvable host name, we defined “dev.mydomain.com” for the Discovery Service in the DNS setup phase)

  • Click “Next”
  • Leave the default for the Internet Facing Server location (“auth.mydomain.com” has been also defined in the DNS setup phase)

The system checks should come with 2 successes.

  • Click the “Apply” button on the summary screen and you reach the Finish screen

Create and configure Relying Party Trusts in ADFS for the IFD Endpoint

Start AD FS Management from the Server Management’s Tools and follow these steps:

  • Right click on “ADFS”
  • Select “Add Relying Party Trust…” on the contextual menu

  • This open the “Add Relying Party Trust Wizard”, click “Start”
  • On the “Select Data Source” page, click the choice “Import data about the relying party published online or on a local network”
  • Type the Federation Metadata URL to locate the federationmetadata.xml file for IFD (should start with the prefix URL you specified in the IFD setup, https and port 444 and then the suffix is the same as for the Internal Federation Metadata URL: https://auth.mydomain.com:444/federationmetadata/2007-06/federationmetadata.xml

  • Click “Next” and on the “Specify Display Name” page, set the Display name as “CRM IFD Relying Party” for example

  • Pass the “Configure Multi-factor Authentication” with a “I do not want to configure …”
  • On the “Choose Issuance Authorization Rules” page, keep “Permit all users to access this relying party” and click “Next”
  • Confirm the page “Ready to Add Trust” by clicking “Next”

  • On the Finish page, leave the tick box “Open the Edit Claim Rules dialog for this relying party trust when the wizard closes” and close the wizard
  • If you missed the previous step, you can right click on the relying party that you created, and click “Edit Claim Rules”
  • On the Claim Rules editor popup, click “Add Rule” and select the rule template “Pass Through or Filter an Incoming Claim”

  • Create the Rule n°1:
    • Claim rule name: Pass Through UPN
    • Incoming claim type: UPN
    • Check “Pass through all claim values”
  • Click “Finish”

  • On the Claim Rule Editor, click the button “Add Rule” again
  • Select the Claim rule template “Pass Through or Filter an Incoming Claim” and click “Next”
  • Create the Rule n°2:
    • Claim rule name: Pass Through Primary SID
    • Incoming claim type: Primary SID
    • Check “Pass through all claim values”
  • Click “Finish”
  • On the Claim Rule Editor, click the button “Add Rule” again
  • Select the Claim rule template “Transform an Incoming Claim” and click “Next”
  • Create the Rule n°3:
    • Claim rule name: Transform Windows Account Name to Name
    • Incoming claim type: Windows account name
    • Outgoing claim type: Name
    • Check “Pass through all claim values”

  • Click “Finish”
  • Back to the Claim Rules editor, check you have your 3 rules and click “Apply” and “OK”

  • You should have 3 Relying Party Trusts in the ADFS Trust Relationships

Change the Port of ADFS with the command line and restart the ADFS Service (adfssrv):

  • Set-adfsProperties -nettcpport 809

Restart IIS with the command line:

  • iisreset

For information, you can browse the following URL and get the ADFS Web Service WSDL:

https://sts.mydomain.com/adfs/services/trust/mex (replace with your adfs name and your domain).

Internet-Facing Deployment Test

Access to Dynamics CRM 2016 from the internet (not from the server you did the install) and use the following URL:

https://{crmOrg}.{mydomain.com}:444

e.g.: https://crm2016.mydomain.com:444

that should lead your browser to this authentication screen:

Login with the following format: domain\username

Congratulation !!! you made it through the IFD for Dynamics CRM 2016.

In case of issue, there are ways to help you troubleshoot it. Your favourite search engine is your best friend. Check out also the links below that helped a lot to complete this installation.

Mobile App (Optional)

Once you got MS Dynamics CRM working with the IFD, you can access it from the Dynamics CRM mobile app.

You can download this app on multiple phones and tablets OS:

CRM for phones app



CRM for tablets app



Once the mobile app started, just enter your external CRM URL, e.g. https://crm2016.mydomain.com:444.

And that’s it, after a short time loading some modules, you should be able to use your CRM from your phone or tablet.

Useful Links

Here is a list of great links that really guided me very well through this installation:

Here are some official documentation links:

Finally, some useful links that helped me for troubleshooting some issues during the installation:

Posted in CRM, Integration, Marketo, Microsoft Dynamics CRM 2016, Technical Architecture

Microsoft Dynamics CRM 2016 On-Premise Deployment on Azure VM – Part 1 – Core Infrastructure and CRM Installation

Objectives

This article in 2 parts, explains step by step how to set up a Microsoft Dynamics CRM 2016 development server on an Azure Virtual Machine and to open it to the Internet via the CRM Internet-Facing Deployment.

These steps are not intended to be followed for a Production server as they rely on a single Virtual Machine for all the infra-structure including:

  • Windows Server 2012 R2 Operating System
  • Active Directory Domain Services role,
  • Web Server (IIS) Role
  • Active Directory Federation Service (ADFS) V3.0 role,
  • SQL Server Enterprise 2014 SP1,
  • and of course Dynamics CRM 2016 with all roles on the same server.

This is clearly not a configuration recommended and supported by Microsoft but it’s very didactic as it touches to a good range of Microsoft server technologies and concepts.

This deployment adds a bit of complexity regarding some required work-arounds to make it run properly, but also remove some complexity in overall (less servers to deploy and configure, lower administration). Some straight lines and assumptions are taken, for example regarding the Windows accounts that again is not suitable for a Production environment.

Part1 covers the core infrastructure installation with MS Dynamics CRM 2016 configured for a simple AD Authentication, restricted to internal users. The schema below highlights the process:

You can complete Part 1 of this article and stop there with a fully functional Dynamics CRM 2016 server accessible only from the server itself.

Part 2 covers the Internet-Facing Deployment configuration of Dynamics CRM 2016 to have it support Claim-Based authentication for internal and external access thanks to the MS Active Directory Federation Service. The schema below highlights the process:

Requirements

For this deployment you need:

  • An Azure account: https://azure.microsoft.com with some credit available,
  • A Dynamics CRM 2016 On-premise license key (ideally through a MSDN subscription like Visual Studio Ultimate or you can use a trial key).

Virtual Machine Provisioning on Azure (new Portal)

2 alternative options are presented here:

  • Option 1 relies on an Azure Virtual Machine provided with both Windows Server 2012 R2 and SQL Server Enterprise 2014 SP1. This option is interesting if you do not have a license of SQL Server but would cost more since it will come with a mandatory storage disk, especially if you go for the premium SSD version
  • Option 2 relies on an Azure Virtual Machine provided with only Windows Server 2012 R2. You’ll have to install your own SQL Server, but it is possible to make it use the VM disk space instead of an independent storage disk, which is fine for a development box.

Option 1: VM with Windows Server 2012 R2 + SQL Server Enterprise 2014 SP1

You need, first of all, a Resource group on the new Azure Portal:

  • You can create it by clicking “New” à “Resource group”:
  • Give it a name, select your Azure subscription and then pick a location that is closest to you

You can now create a new Virtual Machine from the new Azure Portal:

  • Pick the “Database Servers” group and select the “SQL Server 2014 SP1 Enterprise on Windows Server 2012 R2”
  • SQL Server Enterprise is recommended (64 bits), SQL Server Compact or Express are not supported for Dynamics CRM (see Requirements in the links provided on the second article, Part 2)

  • Name your Virtual Machine
  • Define your admin account that will be used for the rest of this article
  • Select your Azure subscription
  • Select your Resource Group

On the next page you need to size your Virtual Machine:

  • Be careful with your choice, this is going to impact your burn rate and overall cost
  • We pick the DS2 Standard with 2 Cores, 7 GB of RAM, a disk of 14 GB that seems comfortable for a development environment

  • On the next page we would recommend the Standard disk type for a development box (the premium one adds a lot to the Azure burn rate)
  • We take the opportunity to create a Storage account, a Virtual Network with a public IP (static), Network Security Group from this interface
  • The Availability Set is not required for this dev. Environment

  • On the next page we define the SQL Server settings:
    • SQL connectivity: Private
    • Port: 1433 by default
    • Optionally Enable SQL Authentication and create a login and password for the SQL authentication (Windows Authentication only is fine)

Once the VM provisioning is complete, select “Connect to open a remote desktop session” and access your VM from your remote PC or Mac with Microsoft Remote Desktop. This software can be installed for free on Windows or iOS.

  • Use your VM username and password to access your Virtual Machine
  • By default, the Remote Desktop access port 3389 is enabled on your VM but be aware that this is the only open port to start with
  • Don’t try to ping your VM with your public IP, Azure doesn’t seem to let it going thru

Option 2: VM with Windows Server 2012 R2 only

Make sure you have a Resource group on the new Azure Portal (see Option 1).

You can now create a new Virtual Machine from the new Azure Portal:

  • Pick the “Windows Server 2012 R2 Datacenter”

  • Follow the same steps as Option 1, but you will not encounter the SQL Server settings.

Now we are going to install our own database software.

Transfer across your VM an installer of Microsoft SQL Server 2014 SP1 Enterprise (or another MS SQL Server version supported by Dynamics CRM 2016) and launch the installer.

  • Select ‘New SQL Server stand-alone installation’

  • Accept the software license terms on the next screen
  • On the Install Rules screen, you can ignore the warnings about the Computer Domain Controller and Windows Firewall, since we are building a development environment

  • On the Setup Roles screen, select ‘SQL Server Feature Installation’ and click ‘Next’
  • On the Feature Selection screen make sure to check the following features:
    • Database Engine Services
    • Full-Text and Semantic Extractions for Search
    • Reporting Services – Native
    • Management Tools – Basic
    • Management Tools – Complete
  • Leave the default installation paths proposed by the wizard

  • On the next screen, you maybe signaled some missing features like the .NET Framework 3.5 Feature. You’ll need to go and add those features via the Server Manager (Manage à Add Roles and Features). You can proceed with the installer afterward if no reboot is required.

  • If no missing features, you’ll reach the Instance Configuration screen
  • Leave the Default instance with its default Instance ID

  • On the Server Configuration screen, associate the following Account Names:
    • SQL Server Agent: NT AUTHORITY\SYSTEM
    • SQL Server Database Engine: NT AUTHORITY\SYSTEM
  • Leave the other accounts as default

  • On the Database Engine Configuration, you can keep the Windows authentication mode or optionally use the mixed mode
    • In that second case, provide a password for the ‘sa’ account
  • Click the button ‘Add Current User’ to add your current admin account as the SQL Server administrator. You can also add later other users such as your CRM Admin user.

  • On the next screen, confirm ‘Install and Configure’
  • And finally, you’ll be ready to install … click ‘Install’

  • Check the Windows services, and make sure the SQL Server Agent is running. If not, you’ll have to change the Log On user in the service properties and use an admin user declared in the Active Directory (like your current user).

Active Directory Domain Services Installation and Configuration

Access your VM via Microsoft Remote Desktop, open the Server Manager:

  • Click “Add Roles and Features”
  • Leave the choice “Role-based or feature-based installation”

  • Select your server from the server pool

  • Add the Server Role “Active Directory Domain Services”
  • Accept to add all the required features for Active Directory Domain Services

  • Make sure to also include these features:
    • Windows Identity Foundation 3.5
    • Windows Search Service

  • Click “Next” button

  • Click “Next” button and accept the Wizard to automatically restart the server if needed

  • Click “Install” and wait the end of the process
  • Click “Promote this server to a domain controller”

  • Check “Add a new forest”
  • Define your Root domain name: e.g. mydomain.com
    • It will be matching the domain we will register with a public internet Domain provider later, so select it carefully and check if this domain is available

  • Click “Next”
  • On the next page, fill up the following fields:
    • Forest functional level: Windows Server 2012 R2
    • Domain functional level: Windows Server 2012 R2
    • Check the box “Domain Name System (DNS) server”
    • Choose a password for the Directory Services Restore Mode (DSRM) and note it somewhere safe

  • Click “Next” to reach the DNS Options
  • Ignore the popup message notifying you that “a delegation for this DNS server cannot be created…” as this is irrelevant for a developer server

  • Click “Next” and verify the NetBIOS domain name assigned to the domain

  • Click “Next”
  • Accept all the default locations for the AD DS database, log files and SYSVOL

  • Click “Next” and make sure the Prerequisites Check is passed

  • You can safely ignore the warnings from the Prerequisites Check
  • Click “Install”
  • The server reboots at the end of the promotion operation, this is fine, you’ll be able to connect again in a few minutes
    • You’ll use your domain credentials instead of your local server account, e.g. mydomain\login

IIS Web Server and Application Server Installation

Open again the Server Manager:

  • Add more server roles:
    • Application Server
    • Web Server (IIS)

  • Click “Next”
  • Leave the selected features

  • Click “Next”
  • On the “Role Services” page for the Application Server, check the following services and add all related features:
    • .Net framework 4.5
    • Web Server (IIS) Support
    • HTTP Activation

  • Click “Next”
  • Select the following Management Tools on the “Role Services” for the Web Server (IIS):
    • IIS Management Console
    • IIS Management Scripts and Tools
    • Management Service

  • Click “Next” and “Install’ then wait the installation process to complete
  • All the new server roles will now appear in the Server Manager

Organizational Unit and CRM Users creation in Active Directory

Now we are going to add in the domain an Organizational Unit and some users for the CRM:

  • From the Server Manager, click the “Tools” menu and select “Active Directory Administrative Center”

  • Create some CRM users, e.g.
    • CRM Administrator: mydomain\crmadmin
    • Better check Account never expired for dev users

  • Add the administrator account into the performance Log Users (it is required if you use this account for the Dynamics CRM set up)

  • Don’t forget to add a password for your users
  • Click “OK” for each user

  • Next, create a new “Organizational Unit”

  • On the Organizational Unit page:
    • Name=CRM2016
    • The CRM setup will add its specific AD security groups in this OU

  • Click “OK” and leave the wizard

Reporting Services Configuration

Reporting Services are already installed on the VM with SQL Server Enterprise 2014 SP1. We just need to perform a few configuration steps:

  • Open the Reporting Services Configuration Manager and connect it to your server
  • Select Report Server Service Account as “Use built-in account” and pick “Local System”
    • Note that the Report Server local account is not supported by Dynamics CRM

à If you went for the option 2 and installed SQL Server yourself, then you can ignore the following steps about the Reporting Services.

  • Click “Apply” and move on to the “Web Service URL” page
  • Just click the “Apply” button to create the IIS directory with default settings on the default web site

  • Click “Apply” and move on the “Database” page
  • Click the “Change Database” button
  • On the popup wizard, select “Create a new report server database” and click “Next”

  • We are done with the SQL Server Reporting Services (SSRS) configuration

Microsoft Dynamics CRM 2016 Installation

Now it’s time to install Microsoft Dynamics CRM 2016 Server. Launch the Setup:

  • Click the link “Install Microsoft Dynamics CRM Server”

  • Get the updates for Dynamics CRM if any
  • Click “Next”

  • Install all the listed required components

  • Click “Next” if all components were installed successfully

  • Choose a path for the installation files, the default is fine
  • Click “Next”

  • Specify the Server Roles
    • We are building a development environment where all Server Roles are deployed on the same server, so check all tick boxes
    • In Production, we would probably dispatch the different roles on multiple servers with some redundancy servers in play
  • Click “Next”

  • On the Deployment Options, select “Create a new deployment”
  • Select the name of the computer that is running SQL Server, the same as the one where you’re installing Dynamics CRM
  • Click “Next”

  • Browse and select the Active Directory Organizational Unit (OU) created earlier
  • Click “Next”

  • On the Service Accounts specification page, we should normally define one specific account per service, but that would be for a Production environment
  • In our case, we’ll keep the default “Network Service” for all services
    • You could use a specific account like the “CRM Admin” you created earlier, but we decided to leave this account to be effectively the CRM Administrator later
  • Click “Next”

  • On the Website page, it is very important in our case to create a new website dedicated to Dynamics CRM
    • The default Web site is reserved for ADFS and in our typical scenario is not recommended to use (even if our version of ADFS doesn’t really rely on IIS, there are still some binding magic going on)
    • Create a new Website with Port Number = 5555
  • Click “Next”

  • We do not plan to deploy the Email Router now (this can be done later)
  • Leave the box blank and click “Next”

  • On the Organization Settings screen, fill up:
    • Display Name: CRM2016 (same as Organizational Unit)
    • Unique Database Name: CRM2016 (same as Organizational Unit)
    • Select your currency
    • Leave SQL Collation by default
  • The selected Organization Name “CRM2016” will be reused in the subdomain of the external URL so choose it carefully
  • It is possible to add other Dynamics CRM Organizations in top of this one later
  • Click “Next”

  • Check that the default Report Server URL is correct (see in Reporting Services configuration)
  • Click “Next”

  • This is time for a full system checks
  • Don’t worry too much about the security warning, there are normal because we are deploying a development environment
  • Click “Next”

  • You should reach out the final screen when the Dynamics CRM Server installation is completed
  • Tick the box “Launch Reporting Extensions for SSRS Setup” so the next wizard can start
  • Click “Finish”

  • IIS should now have the Microsoft Dynamics CRM web site in top of the Default Web Site, you can do this quick check

Reporting Extensions Installation

In order to start the CRM Reporting Extensions setup wizard, you could have ticked the dedicated box on the previous wizard or launch it manually from the Dynamics CRM setup:

  • Check “Get updates for MS Dynamics CRM”
  • Click “Next”

  • Select the Database Server, in our case it is deployed on the same server
  • Click “Next”

  • Select the SSRS Instance name by default
  • Click “Next”

  • Leave the default Installation directory
  • Click “Next”

  • Time for the system checks
  • Again, in the case of our Development server, we are not too worried about the security warnings
  • Click “Next” and start the installation

  • This is the completion screen you see when the installation is done successfully

Dynamics CRM 2016 Testing and Post Admin

Accessing Dynamics CRM 2016 for the first time

You can access to Dynamics CRM 2016 with the url: http://{your_server_name}:5555

  • Do a quick test drive and make sure everything works all right

Sample CRM Data Addition (Optional)

For a Development environment, it’s always a good idea to add some test data. Good news, Dynamics CRM has got a few in store for you:

  • Pick the “Settings” in the main menu
  • Click on “System”à“Data Management”

  • On the next page, select the icon “Sample Data”

  • Go and install the Sample Data

Extra CRM Users Creation (Optional)

  • Click on the main menu “Settings” and pick “Security”

  • Click on “Users” and click the button “+ New”

  • Enter a user name of someone that is already in your Active Directory
    • E.g our CRM Administrator created earlier with user name = crmadmin
    • Data will be autocompleted from AD
  • Click “Save”

  • Once the user is saved, click button “Manage Roles”

  • Typically for our CRM Administrator we select the roles:
    • “Activity Feeds”
    • “Salesperson”
    • “System Administrator”

Posted in CRM, Integration, Marketo, Microsoft Dynamics CRM 2016, Technical Architecture

Integrate seamlessly Marketo with 500+ Apps with Zapier (Practical Use Cases with FullContact and Google Sheets)

Objectives

This article, also published on the Marketo Developer Blog, explains in details how to integrate Marketo with potentially over 500 Cloud Apps, thanks to Zapier.

For that, we’re going to build from scratch a Zapier connector for Marketo and implement two practical integration use cases:

Use case 1: a unidirectional Leads integration from FullContact Card Reader to Marketo

  • Scan any contact’s business card with the FullContact mobile Card Reader app and get a lead automatically created in Marketo.

Use case 2: a bidirectional Leads integration between Marketo and Google Sheets

  • Add an existing lead to a static list in Marketo and find the lead automatically added to your Google Sheet.
  • Modify any lead in your Google Sheet and find the change echoed back to Marketo.

Requirements prior to get started

Sign-up for a free account with Zapier

Zapier is a Webapp Automation Service that let you easily automate tasks between other online apps without the need for programmers or any IT resources. Check their web site for more info:

Zapier supports today more than 500 apps in many different domains such as Marketing, CRM, CMS, Customer Support, Electronic Signature, Forms, etc …

A single integration between one app and another is called a Zap.

Check Zapier’s zapbook for an exhaustive list of supported web apps: https://zapier.com/zapbook/.

Sign up for a free account here: https://zapier.com/sign-up/, you’ll get access to up to 100 tasks/month, 5 zaps, zaps running every 15 minutes. You can of course get much more by subscribing to Zapier’s paid plans (basic, business, business plus, etc…).

Access to a Marketo Instance as Administrator or with a provided API User account

Our Zapier connector will use the Marketo REST API in order to push Leads data to Marketo. In order to use this API, you’ll need an API User and a Custom Service that you can create yourself if you are administrator of your Marketo instance. If not, then an administrator will need to provide those to you. There is also a Webhook to create, only accessible to a Marketo Administrator.

The following documentation from the Marketo Developer Web Site explains step by step how to create the Marketo API User and the Custom Service: http://developers.marketo.com/documentation/rest/custom-service/.

Once you’re done, you should have the following credentials in order to invoke the Marketo REST API:

  • Client Id,
  • Client Secret,
  • No need to get a Token as it would be only valid for a single hour; The Connector will generate tokens for you automatically.

Another information you need from your Marketo instance is the Munchkin Account Id. You can get it from the Munchkin or the Web Services Admin screens. Its pattern looks like this: 000-XXX-000.

Sign-up for a free account with Google

Google Docs, Sheets, and Slides are productivity apps that let you create different kinds of online documents, work on them in real time with other people, and store them in your Google Drive online. Our use case needs a Google Sheet.

The following link covers the different features of Google Docs and leads to the creation of an account with Google: https://www.google.com/docs/about/.

Sign-up for a free account with FullContact

FullContact keeps you fully connected to the people who matter most by pulling in all your contacts and continuously syncing them with changes to social profiles, photos, email signatures, company information, and more. They offer a mobile business cards reader that can scan cards into 250+ Web Apps, including Zapier.

You can sign-up for a free account here: https://app.fullcontact.com/login or eventually subscribe to a premium paid account with more features and capacity.

The mobile app can be downloaded here:

The FullContact Zaps are documented here: https://zapier.com/zapbook/fullcontact/.

Implementation of the Marketo Connector for Zapier

Create the Marketo App

From the Zapier web interface, go to the Developers Portal.

Click the button <+ Add New App> and fill out at minima the Title (e.g ‘Marketo’) and the Description. The logo is optional, but nice to have.

Authentication

In this section we declare the different fields used for the Marketo REST API authentication and the authentication settings.

Create first the following fields:

Label Munchkin Account Id Client Id Client Secret
Key munchkin_account_id client_id client_secret
Type Unicode Unicode Unicode
Required Checked Checked Checked

Edit the ‘Authentication Settings’ as in the following screen:

  • Auth Type: Session Auth
  • Auth Mapping:

    {

    “access_token”:”{{access_token}}”

    }

  • Access Token Placement: Token in Querystring

Zapier proposes a few Authentication Types but typically OAuth V2 would not work with Marketo. Currently the Marketo platform does not offer an Oauth2 authentication similar to Facebook where an end user is forwarded to an authentication page. A Marketo custom service has been created giving access to a client id and client secret. We use the client id and client secret to generate an access token via the REST API authentication endpoint. We can then use this access token to make subsequent requests to the REST API. The token expires after an hour and must be generated again to proceed calling the REST API.

We chose authentication Type = ‘Session Auth’ as it allows us to execute a custom authentication script every time our session token is expired. We’ll see in the section ‘Scripting API’ how to implement this mechanism that can only work with this type of authentication.

Triggers

Zapier Triggers are there to bring data into Zapier. We do not need one for our use cases as we will leverage a Marketo Webhook instead. However, we still need to write a dummy Trigger as a mandatory test for our Marketo connector.

We are going to create a Test Trigger calling the Marketo REST API ‘Get Daily Usage’: http://developers.marketo.com/documentation/rest/get-daily-usage.

Click the button <+ Add New Trigger> to start the wizard and fill-up the following fields (fields not mentioned can be left blank):

Name and Description

  • Name: Test Trigger
  • Key: test_trigger
  • Description: The Test Trigger of the Marketo App
  • Important? Not checked
  • Hide? Checked

Trigger Fields

  • None

Where Data Comes From

  • Data Source: Polling
  • Polling URL:
    https://{{munchkin_account_id}}.mktorest.com/rest/v1/stats/usage.json

Sample Result

  • Leave blank

Click now the button <* Manage Trigger Settings> and set our Test Trigger to be the one we’ll use to verify a user’s auth credentials.

Actions

Zapier Actions are there to send data out from Zapier.

We are going to implement the Create_Update Lead Action calling the Marketo REST API ‘Create/Update Leads’: http://developers.marketo.com/documentation/rest/createupdate-leads/. This Action will allow to create a new lead within Marketo, or if the lead already exists, it will update it with the submitted values. We’ll use the field ’email’ for deduplication.

Click the button <+ Add New Action> to start the wizard and fill-up the following fields (fields not mentioned can be left blank):

Name and Description

  • Name: Create_Update Lead
  • Noun: Lead
  • Key: create-update-lead
  • Description: Create a new lead within Marketo, or if the lead already exists update it with the submitted values
  • Important? Checked
  • Hide? Not Checked

Action Fields

Action Fields are the fields users will map data into. Choose them carefully according to your own needs as they will represent all the data you will be able to update in Marketo. There is an option in Zapier to offer to the end user all fields available in Marketo, but that would induce more code and complexity, not required for a disposable connector.

As an example, we selected the following fields:

Label Key Type Required? Parent key Send in POST by default?
Email Address email Unicode Checked input
Partition Name partitionName Unicode Not Checked Checked
First Name firstName Unicode Not Checked input
Last Name lastName Unicode Not Checked input
Phone Number phone Unicode Not Checked input
Notes Lead_Notes__c Textarea Not Checked input
Called called Unicode Not Checked input

Partition Name is mandatory in our case since our Marketo instance has Lead Partitions in service. It could be omitted otherwise. We separated it from the ‘input’ group so the end-user understand this is not a field to synch.

The field ‘Notes’ comes from a synch between Marketo and Salesforce, do not use it if you don’t have it in your Marketo instance.

The filed ‘Called’ has been created in our Marketo instance, do not use it if you don’t have it in your Marketo instance.

Of course, the goal is to let you pick the fields you need from Marketo. It is recommended to start small and add the extra fields later.

Where to Send Data

  • Action Endpoint URL: https://{{munchkin_account_id}}.mktorest.com/rest/v1/leads.json

Sample Result

  • Leave blank

Scripting API

Zapier’s scripting feature allows to manipulate the requests and responses that are exchanged between your app’s API and Zapier. You can modify HTTP requests just before they are sent and can parse responses before Zapier does anything with them. We need it in order to complete our custom ‘Session Auth’ authentication so it works with Marketo.

More info here: https://zapier.com/developer/documentation/v2/scripting/#intro-to-scripting.

Copy the following code and we’ll go through some explanations later on:


var Zap = {

get_session_info: function(bundle) {

console.log('Entering get_session_info method ...');

var access_token,
access_token_request_payload,
access_token_response;

// Assemble the meta data for our Access Token swap request
console.log('building Request with client_id=' + bundle.auth_fields.client_id + ', and client_secret=' + bundle.auth_fields.client_secret);
access_token_request_payload = {
method: 'POST',
url: 'https://' + bundle.auth_fields.munchkin_account_id + '.mktorest.com/identity/oauth/token',
params: {
'grant_type' : 'client_credentials',
'client_id' : bundle.auth_fields.client_id,
'client_secret' : bundle.auth_fields.client_secret
},
headers: {
'Content-Type': 'application/json', // Could be anything.
Accept: 'application/json'
}
};

// Fire off the Access Token request.
access_token_response = z.request(access_token_request_payload);

// Extract the Access Token from returned JSON.
access_token = JSON.parse(access_token_response.content).access_token;
console.log('New Access_Token=' + access_token);

// This will be mixed into bundle.auth_fields in future calls.
//bundle.auth_fields.access_token=access_token;
return {'access_token': access_token};
},

test_trigger_pre_poll: function(bundle) {

console.log('Entering test_trigger_pre_poll method ...');

bundle.request.params = {
'access_token':bundle.auth_fields.access_token
};

return bundle.request;

},

test_trigger_post_poll: function(bundle) {

console.log('Entering test_trigger_post_poll method ...');

var data = JSON.parse(bundle.response.content);
if ((!data.success)&&((data.errors[0].code=="601")||(data.errors[0].code=="600"))){
console.log('Access Token expired or invalid, requesting new one - data.success=' + data.success + ', data.errors[0].code=' + data.errors[0].code);

throw new InvalidSessionException(); // Calling get_session_info() to regenerate Access Token
}

return JSON.parse(bundle.response.content);
},

create_update_lead_pre_write: function(bundle) {

bundle.request.params = {'access_token':bundle.auth_fields.access_token};
return bundle.request;
},

create_update_lead_post_write: function(bundle) {

var data = JSON.parse(bundle.response.content);
if ((!data.success)&&((data.errors[0].code=="601")||(data.errors[0].code=="600"))){
console.log('Access Token expired or invalid, requesting new one - data.success=' + data.success + ', data.errors[0].code=' + data.errors[0].code);
throw new InvalidSessionException(); // Calling get_session_info() to regenerate Access Token
}
return JSON.parse(bundle.response.content);
}

};

get_session_info method

  • This method is responsible for generating or regenerating an access token calling the Marketo REST API ‘Authentication’: http://developers.marketo.com/documentation/rest/authentication/.
  • It is called every time any ‘post_poll’ methods are encountering an ‘Access Token Expired’ error. An access token is scheduled to expired every 1 hour so this is expected.
  • Action Endpoint URL:
    https://{{munchkin_account_id}}.mktorest.com/identity/oauth/token .

All pre_poll and pre_write methods

  • We must create a ‘pre-poll’ method on any Trigger we have created, in order to modify the HTTP request just before it is sent, so we can add the Marketo Access Token in its parameters.
  • We must create a ‘pre-write’ method on any Action we have created, for the same reason.

All post_poll and post_write methods

  • We must create a ‘post-poll’ method on any Trigger we have created, in order to parse responses before Zapier does anything with them, and eventually intercept ‘Access Token Expired’ error.
  • We must create a ‘post-write’ method on any Action we have created, for the same reason.
  • If such an error has occurred, we throw an InvalidSessionException that will tell Zapier to replay the authentication and execute again the get_session_info method.

Note that you can access the Bundle logs from the Scripting API from the ‘Quick links’ menu on the top right corner of the screen. This is really useful to debug the scripts.

And now it’s time for the fun part…

Use Case 1: Integration of Marketo with FullContact Card Reader

For this integration we’ll create one single Zap from FullContact to Marketo. With this Zap, you’ll be able to scan business cards with the FullContact Mobile Card Reader and push the leads to Marketo.

Zap FullContact à Marketo

From the Zapier Dashboard click the button ‘Make a new Zap’.

Trigger in Zapier

  • Pick the App FullContact
  • Choose FullContact Trigger ‘New Business Card’
  • Connect to your FullContact account
  • Test the FullContact App

Action in Zapier

  • Pick the App Marketo we just created earlier, it should display in Beta
  • Choose Marketo Action ‘Create_Update Lead’
  • Connect to your Marketo account, filling up the authentication parameters (Munchkin Account Id, Client Id, Client Secret)
  • Map the fields from FullContact to Marketo

  • Fill-up eventually a Partition Name where your new leads should go (only if partitions exist in your Marketo Instance)
  • Test the Marketo App
  • Activate your Zap

Make sure you download the business cards Reader from FullContact and activate the Zapier Integration right from your mobile device.

Use Case 2: Integration of Marketo with Google Sheets

For this integration we’ll create two Zaps. One from Marketo to Google Sheets and another one from Google Sheets to Marketo. With this Zap, you’ll be able to synch up some of your leads or contacts between Marketo and a Google Sheet.

Zap Marketo Webhook à Google Sheets

For the first Zap, we don’t rely on a custom connector for Marketo, but we leverage Marketo’s Webhooks and the ‘Webhooks by Zapier’ Trigger.

From the Zapier Dashboard click the button ‘Make a new Zap’.

Trigger Part 1 in Zapier

  • Pick the ‘Webhooks by Zapier’ Trigger App
  • Check ‘Catch Hook’ that will allow to wait for a POST or GET to a Zapier URL
  • No need to pick off a child key
  • Zapier generated a custom webhook URL
    for you to send requests to, copy it in the clipboard

Webhook in Marketo (steps to be done by an Administrator)

  • Go to Admin à Webhooks
  • Create a new Webhook called ‘Push Lead to Zapier’ and edit the Webhook form :

  • In the template’s field, declare all the Lead’s fields you would like to transfer to Zapier and leverage the Marketo’s tokens. For our Use Cases, we take the same fields we defined for the custom Zapier connector that push Leads to Marketo:
{“firstName”:”{{lead.First Name}}”,”lastName”:”{{lead.Last Name}}”,”email”:”{{lead.Email Address}}”,”phone”:”{{lead.Phone Number}}”,”leadOwner”:”{{lead.Lead Owner First Name}} {{lead.Lead Owner Last Name}}”,”leadOwnerEmail”:”{{lead.Lead Owner Email Address}}”,”leadNotes”:”{{lead.Lead Notes:default=edit me}}”,”called”:”{{lead.Called}}”}
  • Save the form
  • No need for a Response Mapping, so you’re done with the webhook

Test Campaign in Marketo (steps to be done by a Marketer or an Administrator)

  • From the Marketing Activities, create a new Smart Campaign

    For testing purpose we are going to create a campaign that trigger our Webhook each time a lead is changing its status to MQL. Of course you can use the webhook for any other business purpose.

  • Edit the Smart List

  • Call the Webhook in the Flow

  • Schedule the Campaign

  • Make sure each lead can run through the flow every time
  • Activate the Smart Campaign

Trigger Part 2 in Zapier

  • In order to complete the ‘Webhooks by Zapier’ Trigger App, we need to fire the Marketo Smart Campaign once and catch the Webhook in Zapier
  • In our test case, we just need to go to Marketo Lead Database, open a lead and change its status to ‘MQL’

Create the spreadsheet in Google Sheets

  • Create a new spreadsheet
  • Create a Worksheet or use the default one
  • Add a column for each field you want to synch from Marketo (the ones declared in the Marketo webhook)

Action in Zapier

  • Pick the App Google Sheets
  • Check the option ‘Create Spreadsheet Row’
  • Connect to your Google Sheets account
  • Select your Google Sheets spreadsheet
  • Select the Worksheet
  • Map all the fields between the ‘Webhooks by Zapier’ Trigger App and Google Sheets:

  • Test the Google Sheets App
  • Activate your Zap

Zap Google Sheets à Marketo

From the Zapier Dashboard click the button ‘Make a new Zap’.

Trigger in Zapier

  • Pick the ‘Google Sheets’ Trigger App
  • Tick the ‘Updated Spreadsheet Row’ that triggers when a new row is added or modified in a spreadsheet
  • Connect to your Google account
  • Select the Spreadsheet you want to trigger from (should be the same one used in the previous Zap) and the Worksheet
  • Set Trigger Column to ‘any_column’
  • Test the Google Sheets App

Action in Zapier

  • Pick the App Marketo we just created earlier, it should display in Beta
  • Choose Marketo Action ‘Create_Update Lead’
  • Connect to your Marketo account, filling up the authentication parameters (Munchkin Account Id, Client Id, Client Secret)
  • Map the fields from Google Sheets to Marketo
  • Fill-up eventually a Partition Name where your new leads should go (only if partitions exists in your Marketo Instance)
  • Test the Marketo App
  • Activate your Zap

Wrap-Up

Here are some axes of improvement for our Marketo connector for Zapier:

  • Adding other Triggers and Actions related to diverse Marketo objects (Lists, Custom Objects, etc …).
  • Instead of hard coding the fields from Marketo, it’s possible to pull dynamically the fields from Marketo but that would require some technical translation work between Marketo and Zapier.
  • Sharing the connector with development team and eventually make it generally available.

Zapier has deployed since a Premium Marketo adapter making our use cases much easier to implement. In any cases, this article could always be leveraged in order to integrate Marketo with Zapier with a free Zapier plan and also to build extreme use cases that would not be always supported by the premium adapter. Typically, thanks to the Zapier Scripting API, it would make it possible to implement a connector capable of running some coded logic prior to update Marketo with data.

We hope you enjoyed this article and that it will help you to be even more successful with Marketo and Zapier. Thank You!

Posted in Digital Marketing, Integration, Marketo, Technical Architecture, Zapier

Integrating Microsoft Dynamics CRM 2011 Online with Java and other non-.Net Clients

Introduction

While the most common style of .Net development for Microsoft Dynamics CRM Online is using the SOAP endpoint with early or late bound types, there are situations where you need to use the Web Services Description Language (WSDL) endpoint directly. This article is intended to provide some understanding how to leverage this light-weight endpoint that can be used from non-.NET languages. It provides some practical examples in Java that could be transposed to any other popular language.

Microsoft Dynamics CRM Online organizations created since July of 2012 are using the Microsoft Office 365 authentication and billing platform. This was a switch from using Microsoft LiveID since the beginning of Microsoft CRM Online in April of 2008. This article is covering the latest Office 365 Authentication scenario that would fit most of the recent CRM online deployments. An article from the Microsoft Developer Network (referenced at the end of this article) is covering the LiveID scenario.

We’ll start to look at the generation of the Microsoft Dynamics CRM Organization web service stub from its WSDL endpoint and then we’ll see how to authenticate a user via the Office 365 platform’s Secure Token Service (STS) and encode the resulting tokens in a security header securing all operations with the Dynamics CRM Organization service. Finally some of the basic Organization service operations will be covered along with some considerations around performances and potential issues.

Prerequisites for the Java Client

You’ll need of course a subscription or trial to Microsoft Dynamics CRM 2011 Online. If you don’t have a Dynamics CRM Org. available, you can test drive it here: http://www.microsoft.com/en-us/dynamics/crm-test-drive.aspx.

Dynamics CRM Organization Web Service Stub

The Microsoft Dynamics CRM Organization web service stub can be generated with the WSDL2Java command, available right from the bin folder of Apache Axis2:

{Path to Axis2 Lib}\axis2-1.6.2\bin> WSDL2java -uri https://{Your CRM Org domain}.crm.dynamics.com/XRMServices/2011/Organization.svc?wsdl -p {Java package for the generated stub source} -s -o {Path for the generated stub source}

If you prefer to use Apache Ant in order to automate the stub generation, the Ant task would look like this in your build.xml file:

<!—Microsoft Dynamics CRM 2011 Organization Stub generation –>
<target name=“gen-organization-stub”>
    <taskdef  name=“axis2-wsdl2java”
               classname=“org.apache.axis2.tool.ant.AntCodegenTask”
               classpathref=“build.classpath”/>
    <axis2-wsdl2java wsdlfilename=https://{Your CRM Org domain}.crm.dynamics.com/XRMServices/2011/Organization.svc?wsdl packageName={Java package for the generated stub source}output={Path for the generated stub source}syncOnly=”true” /></target>

The URI prefix to the Dynamics CRM 2011 Organization Service should look like https://myCRMOrg.crm.dynamics.com/ if your Organization is hosted in North America, https://myCRMOrg.crm4.dynamics.com/ for EMEA and https://myCRMOrg.crm5.dynamics.com/ for APAC.

An example of Java package could be like “com.mycompany.dynamicscrm.integration” and the path for the generated stub source should be right within your Java project source folder.

Note that with the “s/syncOnly” option of the command, the generated stubs will contain exclusively the synchronous invocation methods.

A detailed documentation of the WSDL2Java command can be found here.

The generated source code for the stub should be a set of 9 Java classes:

Generated Organization Service Stub

OrganizationServiceStub.java is the main stub for the Microsoft Dynamics CRM 2011 Organization Service and the other classes are Java exceptions related to the service’s basic operations.

Authentication via the Microsoft Office 365 Secure Token Service (STS)

Typical STS Authentication request

Here is a typical SOAP/HTTPS request to authenticate a user through the Microsoft Office 365 STS with the associated main headers:

<s:Envelope
xmlns:s=http://www.w3.org/2003/05/soap-envelope&#8221;
        xmlns:a=http://www.w3.org/2005/08/addressing&#8221;
        xmlns:u=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;>
    <s:Header>
        <a:Action
s:mustUnderstand=“1”>http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action>
        <a:MessageID>urn:uuid:{Message ID}</a:MessageID>
        <a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo>
        <VsDebuggerCausalityData
xmlns=http://schemas.microsoft.com/vstudio/diagnostics/servicemodelsink&#8221;>
            uIDPo2V68j15KH9PqGf9DWiAfGQAAAAA/Dr1z6qvqUGzr5Yv4aMcdIr9AKDFU7VHn7lpNp0zeXEACQAA</VsDebuggerCausalityData>
        <a:To
s:mustUnderstand=“1”>https://login.microsoftonline.com/RST2.srf</a:To>
        <o:Security
s:mustUnderstand=“1”
                xmlns:o=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd&#8221;>
            <u:Timestamp
u:Id=“_0”>
                <u:Created>{Request Timestamp}</u:Created>
                <u:Expires>{Request Expiry Timestamp}</u:Expires>
            </u:Timestamp>
            <o:UsernameToken
u:Id={Token ID}>
                <o:Username>{User Name}</o:Username>
                <o:Password
                Type=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText&#8221;>
                {User Password}</o:Password>
            </o:UsernameToken>
        </o:Security>
    </s:Header>
    <s:Body>
        <t:RequestSecurityToken
xmlns:t=http://schemas.xmlsoap.org/ws/2005/02/trust&#8221;>
            <wsp:AppliesTo
xmlns:wsp=http://schemas.xmlsoap.org/ws/2004/09/policy&#8221;>
                <a:EndpointReference>

                    <a:Address>urn:crmna:dynamics.com</a:Address>

                </a:EndpointReference>

            </wsp:AppliesTo>

                <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>

        </t:RequestSecurityToken>

    </s:Body>

</s:Envelope>

The Parameters
in red
are the dynamic information to define as variables in every authentication request:

  • {Message ID}: unique request Id, e.g. ‘0d457c4e-8b44-4100-8b7e-085ca6303c7f’,
  • {Request Timestamp} : timestamp of the request (now), e.g. ‘2013-08-17T09:32:27.786Z’,
  • {Request Expiry Timestamp}: timestamp when the request should expire (could be now + 5 minutes), e.g. ‘2013-08-17T09:37:27.786Z’,
  • {Token ID}: unique token Id, e.g. ‘uuid-8c6514f1-8cb5-4c6b-8c0f-e476c7fd7a90-1’,
  • {User Name}: User login, e.g. ‘admin@MyCRMorg.onmicrosoft.com’,
  • {User Password}: User password.

A typical successful authentication answer looks like this:

<?xml
version=“1.0”
encoding=“utf-8”
?>
<S:Envelope
xmlns:S=http://www.w3.org/2003/05/soap-envelope&#8221;

        xmlns:wsse=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd&#8221;

        xmlns:wsu=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;

        xmlns:wsa=http://www.w3.org/2005/08/addressing&#8221;>
    <S:Header>
        <wsa:Action
xmlns:S=http://www.w3.org/2003/05/soap-envelope&#8221;

                xmlns:wsa=http://www.w3.org/2005/08/addressing&#8221;

                xmlns:wsu=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;

                wsu:Id=“Action”
S:mustUnderstand=“1”>http://schemas.xmlsoap.org/ws/2005/02/trust/RSTR/Issue</wsa:Action>
        <wsa:To
    xmlns:S=http://www.w3.org/2003/05/soap-envelope&#8221;

                xmlns:wsa=http://www.w3.org/2005/08/addressing&#8221;

                xmlns:wsu=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;

                wsu:Id=“To”
S:mustUnderstand=“1”>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:To>
        <wsse:Security
S:mustUnderstand=“1”>
            <wsu:Timestamp
xmlns:wsu=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;

                     wsu:Id=“TS”>
                <wsu:Created>2013-08-17T09:33:00Z</wsu:Created>
                <wsu:Expires>2013-08-17T09:38:00Z</wsu:Expires>
            </wsu:Timestamp>
        </wsse:Security>
    </S:Header>
    <S:Body>
        <wst:RequestSecurityTokenResponse
    xmlns:S=http://www.w3.org/2003/05/soap-envelope&#8221;

                    xmlns:wst=http://schemas.xmlsoap.org/ws/2005/02/trust&#8221;

                    xmlns:wsse=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd&#8221;

                    xmlns:wsu=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;

                    xmlns:saml=“urn:oasis:names:tc:SAML:1.0:assertion”

                    xmlns:wsp=http://schemas.xmlsoap.org/ws/2004/09/policy&#8221;

                    xmlns:psf=http://schemas.microsoft.com/Passport/SoapServices/SOAPFault&#8221;>

            <wst:TokenType>urn:oasis:names:tc:SAML:1.0</wst:TokenType>

            <wsp:AppliesTo
xmlns:wsa=http://www.w3.org/2005/08/addressing&#8221;>

                <wsa:EndpointReference><wsa:Address>urn:crmna:dynamics.com</wsa:Address></wsa:EndpointReference>

            </wsp:AppliesTo>

            <wst:Lifetime>

                <wsu:Created>2013-08-17T09:33:00Z</wsu:Created>

                <wsu:Expires>2013-08-17T17:33:00Z</wsu:Expires>

            </wst:Lifetime>

            <wst:RequestedSecurityToken>

                <EncryptedData
    xmlns=http://www.w3.org/2001/04/xmlenc#&#8221;

                            Id=“Assertion0”

                            Type=http://www.w3.org/2001/04/xmlenc#Element&#8221;>

                    <EncryptionMethod
Algorithm=http://www.w3.org/2001/04/xmlenc#tripledes-cbc&#8221;>

                    </EncryptionMethod>

                    <ds:KeyInfo
xmlns:ds=http://www.w3.org/2000/09/xmldsig#&#8221;>

                        <EncryptedKey>

                            <EncryptionMethod
Algorithm=http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p&#8221;>

                            </EncryptionMethod>

                            <ds:KeyInfo
Id=“keyinfo”>

                                <wsse:SecurityTokenReference>

                                    <wsse:KeyIdentifier
EncodingType=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary&#8221;

                                                 ValueType=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509SubjectKeyIdentifier&#8221;>

                                        D3xjUG3HGaQuKyuGdTWuf6547Lo=

                                    </wsse:KeyIdentifier>

                                </wsse:SecurityTokenReference>

                            </ds:KeyInfo>

                            <CipherData>

                                <CipherValue>{Security Token 0}</CipherValue>

                            </CipherData>

                        </EncryptedKey>

                    </ds:KeyInfo>

                    <CipherData>

                        <CipherValue>{Security Token 1}</CipherValue>

                    </CipherData>

                </EncryptedData>

            </wst:RequestedSecurityToken>

            <wst:RequestedAttachedReference>

                <wsse:SecurityTokenReference>

                    <wsse:KeyIdentifier

                        ValueType=http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.0#SAMLAssertionID&#8221;>

                        {Key Identifier}

                    </wsse:KeyIdentifier>

                </wsse:SecurityTokenReference>

            </wst:RequestedAttachedReference>

            <wst:RequestedProofToken>

                <wst:BinarySecret>/NbucQb9wbn4h2sZZDpRzwsF8q88SeyP</wst:BinarySecret>

            </wst:RequestedProofToken>

        </wst:RequestSecurityTokenResponse>

    </S:Body>

</S:Envelope>

We can extract from this request the following credentials: the Security Token 0, the Security Token 1 and the Key Identifier.

A typical failed authentication answer, because of invalid credentials, looks like this:

<?xml
version=“1.0”
encoding=“utf-8”
?>

<S:Envelope
xmlns:S=http://www.w3.org/2003/05/soap-envelope&#8221;

        xmlns:wsse=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd&#8221;

        xmlns:wsu=http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;

        xmlns:wst=http://schemas.xmlsoap.org/ws/2005/02/trust&#8221;

        xmlns:psf=http://schemas.microsoft.com/Passport/SoapServices/SOAPFault&#8221;>

    <S:Body>

        <S:Fault>

            <S:Code>

                <S:Value>S:Sender</S:Value>

                <S:Subcode>

                    <S:Value>wst:FailedAuthentication</S:Value>

                </S:Subcode>

            </S:Code>

            <S:Reason><S:Text
xml:lang=“en-US”>Authentication Failure</S:Text></S:Reason>

            <S:Detail><psf:error>

                <psf:value>0x80048821</psf:value>

                <psf:internalerror>

                    <psf:code>0x80041012</psf:code>

                    <psf:text>The entered and stored passwords do not match.</psf:text>

                </psf:internalerror>

            </psf:error></S:Detail>

        </S:Fault>

    </S:Body>

</S:Envelope>

Implementation in Java

First create a template with the full authentication SOAP request’s envelope and tokenize the dynamic variables (tokens are defined with %S):

// SOAP envelope template for MSDC Online STS authentication (O365 online platform)
static
public
final String MSDC_ONLINE_AUTH_SOAP_ENVELOPE_TEMPLATE =
“<?xml version=\”1.0\” encoding=\”UTF-8\”?>” +
“<s:Envelope xmlns:s=\”http://www.w3.org/2003/05/soap-envelope\” “ +
“xmlns:a=\”http://www.w3.org/2005/08/addressing\” “ +
“xmlns:u=\”http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\”>” +
” <s:Header>” +
” <a:Action s:mustUnderstand=\”1\”>http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action>&#8221; +
” <a:MessageID>urn:uuid:%s</a:MessageID>” +
” <a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo>&#8221; +
” <VsDebuggerCausalityData xmlns=\”http://schemas.microsoft.com/vstudio/diagnostics/servicemodelsink\”>uIDPo2V68j15KH9PqGf9DWiAf
GQAAAAA/Dr1z6qvqUGzr5Yv4aMcdIr9AKDFU7VHn7lpNp0zeXEACQAA</VsDebuggerCausalityData>” +
” <a:To s:mustUnderstand=\”1\”>https://login.microsoftonline.com/RST2.srf</a:To>&#8221; +
” <o:Security s:mustUnderstand=\”1\” xmlns:o=\”http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\”>” +
” <u:Timestamp u:Id=\”_0\”>” +
” <u:Created>%s</u:Created>” +
” <u:Expires>%s</u:Expires>” +
” </u:Timestamp>” +
” <o:UsernameToken u:Id=\”%s\”>” +
” <o:Username>%s</o:Username>” +
” <o:Password Type=\”http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText\”>
%s</o:Password>” +
” </o:UsernameToken>” +
” </o:Security>” +
” </s:Header>” +
” <s:Body>” +
” <t:RequestSecurityToken xmlns:t=\”http://schemas.xmlsoap.org/ws/2005/02/trust\”>” +
” <wsp:AppliesTo xmlns:wsp=\”http://schemas.xmlsoap.org/ws/2004/09/policy\”>” +

” <a:EndpointReference>” +

” <a:Address>urn:crmna:dynamics.com</a:Address>” +

” </a:EndpointReference>” +

” </wsp:AppliesTo>” +

” <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>&#8221; +

” </t:RequestSecurityToken>” +

” </s:Body>” +

“</s:Envelope>”;

Generate the dynamic token values and prepare the SOAP request in order to authenticate a user through the Microsoft Online STS:

// Prepare input parameter for CRM Authentication Request
// > Random Message Id
String paramMessageId = UUID.randomUUID().toString();
// > Request Timestamp and +5 minutes validity
TimeZone gmtTZ = TimeZone.getTimeZone(“GMT”);
SimpleDateFormat formatter = new SimpleDateFormat(“yyyy-MM-dd’T’HH:mm:ss”);
formatter.setTimeZone(gmtTZ);
Calendar calendar = Calendar.getInstance(gmtTZ);
Date timestampRequest = calendar.getTime();
calendar.add(Calendar.MINUTE, 5);
Date timestampExpiryRequest = calendar.getTime();
String paramTimestampRequest = formatter.format(timestampRequest);
String paramTimestampExpiryRequest = formatter.format(timestampExpiryRequest);
// > Random Token Id
String paramTokenId = “uuid-“ + UUID.randomUUID().toString() + “-1”;
// Prepare CRM Online authentication SOAP request
String onlineCRMAuthSOAPEnvelope = String.format(
    MSDC_ONLINE_AUTH_SOAP_ENVELOPE_TEMPLATE,
    paramMessageId,
    paramTimestampRequest,
    paramTimestampExpiryRequest,
    paramTokenId,
    userName,
    userPassword);

Now it is time to send the authentication request to the Microsoft Online STS, via SOAP/HTTPS …

// Send CRM Online authentication SOAP request to Microsoft online STS
String onlineCRMAuthResponseXML = postSOAPRequest(
    “https://login.microsoftonline.com/RST2.srf“,
    onlineCRMAuthSOAPEnvelope);

… For that, we can leverage the Apache HTTP Components:

public
static String postSOAPRequest(URI serviceUri, String soapEnvelope) throws ParseException, IOException {
    HttpResponse response = null;
    HttpParams params = new BasicHttpParams();
    params.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 180000); //time out in ms
    HttpClient client = new DefaultHttpClient(params);
HttpPost post = new HttpPost(serviceUri);
StringEntity entity = new StringEntity(soapEnvelope);
post.setHeader(“Content-Type”, “application/soap+xml; charset=UTF-8”);
post.setEntity(entity);
response = client.execute(post);

return EntityUtils.toString(response.getEntity());
}

Finally, parse the SOAP response from the STS and gather 3 credentials: securityToken0, securityToken1
and keyIdentifier.

If the credentials cannot be found, then we assume we are facing a failed authentication answer. In that case, the reason and the detail of the error can be parsed instead:

// Parse the CRM Online authentication SOAP response from STS
// Create a Java DOM XML Parser
DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
// Parse XML with Java DOM XML Parser
Document xmlDocument = builder.parse(new ByteArrayInputStream(onlineCRMAuthResponseXML.getBytes()));
            
// Retrieve security tokens and key identifier from security token response.
XPath xpath = XPathFactory.newInstance().newXPath();
String securityToken0 = readStringValFromXmlDocument(xmlDocument, “//*[local-name()=’CipherValue’]”,xpath);

// If first token is blank, search eventual authentication failure message
if((securityToken0==null)||(securityToken0.isEmpty())){
    String errorReason = readStringValFromXmlDocument(xmlDocument, “//*[local-name()=’Reason’]”,xpath);
    String errorDetail = readStringValFromXmlDocument(xmlDocument, “//*[local-name()=’Detail’]”,
        xpath).substring(20);
            
    if((errorReason!=null)&&(errorReason.equalsIgnoreCase(“Authentication Failure”))){
        logger.debug(“Failed authentication for User ‘” + userName + “‘. Reason is ‘” + errorReason +
                            “‘ and Detail is “ + errorDetail);
        throw
new MSDCAuthenticationException(errorDetail); // Exception to craft according to your needs
    } else {
        logger.debug(“Failed authentication for User ‘” + userName + “‘ but cannot parse the reasons”);
        throw
new MSDCAuthenticationException (“Failed authentication for unexpected reasons);
    }
}
String securityToken1 = readStringValFromXmlDocument(xmlDocument, “(//*[local-name()=’CipherValue’])[2]”,xpath);
String keyIdentifier = readStringValFromXmlDocument(xmlDocument, “//*[local-name()=’KeyIdentifier’]”, xpath);

Implementation for the method readStringValFromXmlDocument: 

public static String readStringValFromXmlDocument(Document xmlDocument, String xpathQueryExpression, XPath xpathInstance) throws XPathExpressionException {return xpathInstance.compile(xpathQueryExpression).evaluate(xmlDocument);
}

Calling Dynamics CRM Organization Service Invocation Methods in Java

Instantiation of the Organization Service Stub

In order to instantiate the Organization Service Stub from the Java classes generated from the WSDL, we must first configure Axis2 by creating a configuration context from the ‘axis2.xml’ file that we placed at the root of the project source code. This file is a copy of the standard ‘axis2.xml’ file that can be found in the ‘conf’ directory of the Axis2 Java library.

We need to also pass, along with the configuration context, the URL of the Dynamics CRM 2011 Online Organization.

// Create OrganizationServiceStub
String fileSeperator = System.getProperty(“file.separator”);
String userDir = System.getProperty(“user.dir”);
String axis2ConfigFilePath = userDir + fileSeperator + “src” + fileSeperator + “axis2.xml”;
                
ConfigurationContext ctx = ConfigurationContextFactory.createConfigurationContextFromFileSystem(userDir, axis2ConfigFilePath);
organizationServiceStub = new OrganizationServiceStub(ctx, https://myCRMOrg.crm.dynamics.com);
                
// Get service client implementation used by this stub.
serviceClient = organizationServiceStub._getServiceClient();

We keep most of the default parameters in the ‘axis2.xml’ file except a custom inflow predefined Phase called ‘MustUnderstandChecker’ that must be declared in this file:

. . .
<!– ================================================= –>
<!– Phases –>
<!– ================================================= –>
<phaseOrder
type=“InFlow”>
. . .            
    <phase
name=“RMPhase”/>
    <!– System predefined phases –>
    <!– After Postdispatch phase module author or service author can add any phase he want –>
     <phase
name=“OperationInPhase”>
    <handler
name=MustUnderstandChecker
    class=
com.mycompany.dynamicscrm.integration.Axis2MustUnderstandChecker>
        <order
phase=“OperationInPhase”/>
    </handler>
    </phase>
    . . .
</phaseOrder>
. . .

Here is an implementation of the Axis2MustUnderstandChecker class that tells Axis2 client to process the security SOAP header block from the message context header:

/***
* Handler for SOAP header.
*
*/
public
final
class Axis2MustUnderstandChecker extends AbstractHandler {

public Axis2MustUnderstandChecker() {
}

/* (non-Javadoc)
* Process the Security SOAP header block from the message context header.
* @see org.apache.axis2.engine.Handler#invoke(org.apache.axis2.context.MessageContext)
*/

public InvocationResponse invoke(MessageContext msgContext)

throws AxisFault {
SOAPHeader header = msgContext.getEnvelope().getHeader();

if (header != null) {
Iterator<?> blocks = header.examineAllHeaderBlocks();

while (blocks.hasNext()) {
SOAPHeaderBlock block = (SOAPHeaderBlock) blocks.next();

if(block != null){     
    if (block.getLocalName().equals(“Security”)) {
        block.setProcessed();
    }
}
}
}

return InvocationResponse.CONTINUE;
}
}

SOAP Requests options and Security SOAP header block

After the instantiation of the Organization Service Stub, some options must be set and a fresh Security SOAP header block must be defined:

try {
    Options scOptions = serviceClient.getOptions();
    scOptions.setMessageId(“urn:uuid:” + UUID.randomUUID().toString());
    EndpointReference endPoint = new EndpointReference(http://www.w3.org/2005/08/addressing/anonymous&#8221;);
    scOptions.setReplyTo(endPoint);
    serviceClient.setOptions(scOptions);
            
    // Add fresh Security SOAP Header block
    serviceClient.addHeader(generateFreshSecuritySoapHeaderBlock(securityHeader));
    serviceClient.engageModule(“addressing”);
            
} catch (AxisFault af) {
            throw
new MSDCIntegrationException(“Unexpected web service error”, af);
}

Here is the detail of the method ‘generateFreshSecuritySoapHeaderBlock’
that generate a brand new security SOAP header block including some fresh timestamps with a validity period (set arbitrarily in our case to 5 minutes) and a security header embedding the credentials from the Microsoft Online STS authentication:

private SOAPHeaderBlock generateFreshSecuritySoapHeaderBlock (String securityHeaderStr) throws XMLStreamException {
        
    SOAPHeaderBlock securitySoapHeaderBlock = null;
    OMFactory omFactory = OMAbstractFactory.getOMFactory();
    OMNamespace securitySecextNS = omFactory.createOMNamespace(http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd&#8221;, “o”);
    OMNamespace securityUtilityNS = omFactory.createOMNamespace(http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd&#8221;, “u”);
    
    // Create fresh Time stamp element for the SOAP header block
    TimeZone gmtTZ = TimeZone.getTimeZone(“GMT”);
    SimpleDateFormat formatter = new SimpleDateFormat(“yyyy-MM-dd’T’HH:mm:ss”);
    formatter.setTimeZone(gmtTZ);
    Calendar calendar = Calendar.getInstance(gmtTZ);
    Date timestampRequest = calendar.getTime();
    calendar.add(Calendar.MINUTE, 5);
    Date timestampExpiryRequest = calendar.getTime();
    String timestampRequestStr = formatter.format(timestampRequest);
    String timestampExpiryRequestStr = formatter.format(timestampExpiryRequest);
    
    OMElement timeStampElement = omFactory.createOMElement(“Timestamp”, securityUtilityNS);
    timeStampElement.addAttribute(“Id”, “_0”, securityUtilityNS);
    OMElement createdElement = omFactory.createOMElement(“Created”, securityUtilityNS);
    OMText createdTime = omFactory.createOMText(timestampRequestStr + “Z”);
    createdElement.addChild(createdTime);
    OMElement expiresElement = omFactory.createOMElement(“Expires”, securityUtilityNS);
    OMText expiresTime = omFactory.createOMText(timestampExpiryRequestStr + “Z”);
    expiresElement.addChild(expiresTime);
    timeStampElement.addChild(createdElement);
    timeStampElement.addChild(expiresElement);

    // Create the Security SOAP header block and add, as a child, Time stamp element

    securitySoapHeaderBlock = OMAbstractFactory.getSOAP12Factory().createSOAPHeaderBlock(“Security”, securitySecextNS);

    securitySoapHeaderBlock.setMustUnderstand(true);

    securitySoapHeaderBlock.addChild(timeStampElement);

    securitySoapHeaderBlock.addChild(AXIOMUtil.stringToOM(omFactory, securityHeaderStr));

    

    return securitySoapHeaderBlock;

}

The method ‘generateFreshSecuritySoapHeaderBlock’ takes the Security Header string as an input parameter and it can be built with the following template:

// Security header template
static
public
final String MSDC_SECURITY_HEADER_TEMPLATE =
“<EncryptedData xmlns=\”http://www.w3.org/2001/04/xmlenc#\” Id=\”Assertion0\” Type=\”http://www.w3.org/2001/04/xmlenc#Element\”>” +
”    <EncryptionMethod Algorithm=\”http://www.w3.org/2001/04/xmlenc#tripledes-cbc\”/>” +
”    <ds:KeyInfo xmlns:ds=\”http://www.w3.org/2000/09/xmldsig#\”>” +
”        <EncryptedKey>” +
”            <EncryptionMethod Algorithm=\”http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p\”/>” +
”            <ds:KeyInfo Id=\”keyinfo\”>” +
”                <wsse:SecurityTokenReference xmlns:wsse=\”http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\”>” +
”                    <wsse:KeyIdentifier EncodingType=\”http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary\” “ +
”                        ValueType=\”http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509SubjectKeyIdentifier\”>%s</wsse:KeyIdentifier>” +
”                </wsse:SecurityTokenReference>” +
”            </ds:KeyInfo>” +
”            <CipherData>” +
”                <CipherValue>%s</CipherValue>” +
”            </CipherData>” +
”        </EncryptedKey>” +
”    </ds:KeyInfo>” +
”    <CipherData>” +
”        <CipherValue>%s</CipherValue>” +
”    </CipherData>” +
“</EncryptedData>”;

Generate the security header string with the following Java code, replacing the 3 tokens in the template by the credentials obtained from the Microsoft Online STS authentication:

// Generate security header
securityHeader = String.format(
    MSDC_SECURITY_HEADER_TEMPLATE,
    keyIdentifier,
    securityToken0,
    securityToken1);

Stub Invocation Methods

Here is an example of a high level ‘Create’ method that create an object in Dynamics CRM 2011 Online with the following parameters:

  • entityLogicalName: the logical name of the entity to create, e.g. ‘account’,
  • attributesHash: an HashMap of attributes (pairs of attribute keys and values), e.g. {“name”, “Test Account”},{“address1_city”, “Paris”}, etc.,
  • organizationService: the Dynamics CRM Organization Service stub instantiated and augmented with the options and the fresh Security SOAP header block.
public String executeCreate(String entityLogicalName, HashMap<String,String> attributesHash, OrganizationServiceStub organizationService) throws MSDCIntegrationException {
        
    // Transfer attributes from HashMap to Stub attribute collection
    OrganizationServiceStub.AttributeCollection attributeCollection = new OrganizationServiceStub.AttributeCollection();
    Iterator<String> it = attributesHash.keySet().iterator();
    while(it.hasNext()){
        String key = it.next();
        OrganizationServiceStub.KeyValuePairOfstringanyType KeyValuePair =
            new OrganizationServiceStub.KeyValuePairOfstringanyType();
         KeyValuePair.setKey(key);
     KeyValuePair.setValue(attributesHash.get(key));
     attributeCollection.addKeyValuePairOfstringanyType(KeyValuePair);
    }
    // Create Entity with attributes
    OrganizationServiceStub.Entity entity = new OrganizationServiceStub.Entity();
entity.setLogicalName(entityLogicalName);
entity.setAttributes(attributeCollection);
OrganizationServiceStub.Create createEntity = new OrganizationServiceStub.Create();
createEntity.setEntity(entity);

// Send Create command to Organization web service
String resultGuid = null;

try {
    OrganizationServiceStub.CreateResponse createResponse;
        createResponse = organizationService.create(createEntity);
        OrganizationServiceStub.Guid createResultGuid = createResponse.getCreateResult();
        resultGuid = createResultGuid.getGuid();    
    } catch (RemoteException
| IOrganizationService_Create_OrganizationServiceFaultFault_FaultMessage e) {
        throw
new MSDCIntegrationException(“Unexpected web service error”,
e);

    }

    logger.debug(“Entity ‘” + entityLogicalName + “‘ created successfully with GUID = “ + resultGuid);


return resultGuid;

}

Here are some of the other public methods you can invoke on the Organization Service stub:

  • Associate: Creates a link between records.
  • Delete: Deletes a record.
  • Disassociate: Deletes a link between records.
  • Execute: Executes a message in the form of a request, and returns a response.
  • Retrieve: Retrieves a record.
  • RetrieveMultiple: Retrieves a collection of records.
  • Update: Updates an existing record.

Performance considerations

Instantiating the Organization Service stub can takes up to 75% of the time of a single transaction, this is why this operation should be minimized at any cost.

It is recommend to keep the same stub instance with one security header for a few consecutive operations within a single stateless transaction. The security SOAP header block has got a timestamp with a validity period and this period should not expire.

If the application is stateful, the stub can be cached. Once instantiated, the security SOAP header block can be regenerated for each further usages. The cost of this operation is minimal.

Before to rebuild a security SOAP header block, the headers of the stub’s service client must be cleared with the command ‘removeHeaders()’:

// Recycling organizationServiceStub
serviceClient = organizationServiceStub._getServiceClient();
// Remove existing headers in order to produce a fresh one
serviceClient.removeHeaders();

Troubleshooting

  • Always make sure that the account used to authenticate on the Dynamics CRM server is valid and that it has the required privileges to execute the desired operations on the Organization service.
  • Undeclared namespace prefix “wsx” Exception at Java runtime when instantiating OrganizationServiceStub:
    • Exception detail: org.apache.neethi.builders.converters.ConverterException: com.ctc.wstx.exc.WstxParsingException: Undeclared namespace prefix “wsx”
    • Probable cause: the generated stub is declaring the “wsx” domain for some tags that are already embedded within some tags from the same domain and the domain is locally declared and not with a global prefix.
    • Proposed solution: edit the generated Organization Service stub “OrganizationServiceStub.java”, search and replace:
      • wsx:MetadataReference by MetadataReference
      • wsx:MetadataSection by MetadataSection
  • The code samples exposed in this article have been tested only with Dynamics CRM 2011 Online (version 5.0.9690.5010, DB 5.0.9690.3417) and not yet with Dynamics CRM 2013. Some changes will need to be implemented once this new version is officially released.

Conclusion

This article has demonstrated a reliable and quite easy to implement way to integrate Dynamics CRM 2011 Online with Java. The different network frames and algorithms have been detailed so they can be adapted with other languages.

The codes samples of this article have been tested in development but in no mean on a production environment. So prior to implement it in production, adequate testing should be considered and this project will be at your own risks.

The next steps would be to push further this approach to Dynamics CRM On Premise:

  • Internet Facing Deployment with ADFS 2.0
  • Windows authentication

References

Source Code on GitHub

You’ll find a full implementation of the approach explained in this article on the following GitHub repository: https://github.com/pdellecase/CRMJavaConnect

MSDN CoE Blog

This article is also published on the Microsoft Dynamics Center of Excellence Blog : http://blogs.msdn.com/b/dynamics-coe/archive/2013/09/21/integrating-microsoft-dynamics-crm-2011-online-with-java-and-other-non-net-clients.aspx

Microsoft Developer Network article: Connect Using Java for Microsoft Dynamics CRM Online

This great article walks you through a basic sample code to integrate Dynamics CRM 2011 Online with Java. The authentication is through LiveID and would not work anymore for recently provisioned Dynamics CRM 2011 Online Organizations.

At the beginning, I was trying hard to generate the Microsoft Dynamics CRM Organization Service stubs with the Java API for XML Web Services (JAX-WS), without any success. This article put me on the right track with the Apache Axis2 library that can parse the Organization Service WSDL without any fatal errors.

Link to the MDN article: http://msdn.microsoft.com/en-us/library/jj602979.aspx

Microsoft Developer Network article: Download the Endpoints Using the Dynamics CRM Developer Resources Page

This article explains how to download the WSDL from the developer resources pages in the Microsoft Dynamics CRM 2011 and Microsoft Dynamics CRM Online Web application. It provides also some details about the Discovery Service and the Organization Service WSDLs.

Link to the MDN article: http://msdn.microsoft.com/en-us/library/gg309401.aspx

Microsoft Developer Network article: IOrganizationService Interface > IOrganizationService Members

This article list the public methods members exposed by the IorganizationService interface, with all their parameters, and is a good starting point to find practical samples of code.

Link to the MDN article: http://msdn.microsoft.com/en-us/library/microsoft.xrm.sdk.iorganizationservice_members.aspx

The Microsoft Dynamics CRM 2011 Software Development Kit (SDK)

The official SDK for Dynamics CRM provided by Microsoft contains many useful .Net samples to get started. I have used some of these samples to monitor, with a tool like Fiddler, the HTTPS traffic between my PC and Microsoft Dynamics CRM in order to gather the authentication request sent to the Office 365 Secure Token Service and the security header from the SOAP requests.

Link to the Microsoft Dynamics CRM 2011 SDK Documentation: http://msdn.microsoft.com/en-us/library/hh547453.aspx

Link to the Microsoft Dynamics CRM 2011 SDK: http://www.microsoft.com/en-us/download/details.aspx?id=24004

Tagged with: , , , , , , , , , , ,
Posted in CRM, Integration, Java, Microsoft Dynamics CRM 2011, Technical Architecture
%d bloggers like this: