How does floating license checkout work?

At a high level, the Intel® floating license checkout process is simple:

1. Check out a seat when the compiler is invoked
2. Check in the seat when the compile is finished

However, a lot happens between steps 1 and 2 that can cause slow checkouts or even denials. 

This flowchart describes the license checkout process for floating licenses at a more detailed level. On the left are the items that run on the client, and the right includes the server process. Some of the items are numbered for more explanation:

1.  First the client builds a list of license files and/or servers to check. Most of the time it will be one or more license files in a default location, and sometimes it will be specified in port@host format in the INTEL_LICENSE_FILE environment variable. This step is repeated until either a license is checked out or there are no more licenses to check. 
2. Once license server data is found, the compiler will attempt to connect to the server on the port and hostname provided. Note: this flowchart excludes any steps for validating named-user/non-floating licenses.
3. If a successful connection is made, the license server will send back the port number for the vendor requested. For Intel® licenses this is typically 28519. The client will then send the checkout information to the vendor daemon.
4. When the vendor daemon receives a request for a license, it will go through all the serial numbers it was launched with until it successfully checks out a license. If there are no more serial numbers to check, it will return an error message to the client.
5. At this point the server must determine if it has a valid license based the data sent by the client. This data includes the feature code, OS info, and product build date.
6. If a valid license exists, the server will determine whether the user needs to check out a license or already has one checked out, based on the username and hostname sent by the client. Intel® licenses allow users to use multiple features without consuming multiple seats. There are some caveats:
   1. If the user is checking out a license from different nodes, such as by compiling over a cluster, it will consume a seat for each unique hostname.
   2. If the user is using features from different licenses, such as running the C++ and Fortran compilers simultaneously, it will consume multiple seats if there are separate serial numbers for C++ and Fortran.
   3. Running a parallel build with a Linux compiler on the same host will consume a seat for each node/CPU used.
7. If the license is valid but there are no seats available, the process will continue to the next serial number (if any). Otherwise, the seat will be consumed by the user for the given hostname and the server will return a success message to the client.
8. If the vendor daemon has checked all available serial numbers but was unable to grant a license, it will return an error message to the client. The client will then continue through its list of licenses until it succeeds or runs out.

The Intel® Software License Manager records detailed information on license checkouts in the log file provided when the license manager (lmgrd) was launched. This log file can be found in the following default locations:

• Linux*: /opt/intel/licenseserver/lmgrd.log
• Windows*: C:\Program Files\Intel\licenseserver\Iflexlm.log

On the client side, detailed logging can be enabled by setting environment variable INTEL_LMD_DEBUG to a file name. Note that enabling client logging can slow license checkouts, and it is recommended to only set it when debugging is necessary.