Cluster failing connections with SSL error message

We recently migrated our production environment from a win2k3/SQL 2000 EE cluster to a new 64 bit win2k3/ SQL 2005 SP1 Cluster.

Cluster works fine for a while (3-6 hours), then our logins start failing. We recieve the following error in the event log:

"The server was unable to load the SSL provider library needed to log in; the connection has been closed. SSL is used to encrypt either the login sequence or all communications, depending on how the administrator has configured the server. See Books Online for information on this error message"

The only way to resume connectivity is to offline and online the sql service in the cluster.

The only reference to this error that I can find has something to do with a hotfix that might be available?

It's sad to say but we are experiencing the same issue.
Only we get this after a good 30 minutes...

Someone from Microsoft Spain is flying over here.

Very painful

|||

In working with microsoft PSS, we found the issue to be a permissions issue of all things.

Our service account did not have rights to lock pages in memory on either machines in the cluster. (even though is a domain administrator).

Once we gave it this right and restarted the SQL Server object in the cluster we have not had a similar failure since.

|||

This was unfortunately enabled already so it did not cause the problem here :-(

|||Hi Wesley,

Any joy with this? I'm currently going throught the same thing, and also have locked pages enabled for my SQL service account. I'm currently going through MS Support, but seem to be going around the houses :o(

Thanks

Stu|||Any luck, we have a similar problem.
We noticed that memory usage grows slowly beyond physical memory (32G) then connection failiurs start and server becomes inoperable.
Workaround solution is to reduce max mem in the server, down to 18G from (25).
This causes the server and OS to release memory. After the system stabilized we increased the maxmem in the server back to its original settng. Lock pages in memory is set.

Hope this helps.

This is a workaround, so we don't have to bounce the server, we are still looking for the cause.
esk.

Setup:2 node Cluster 4*dual 64bit AMD, 32Gb, SQL 2005 Enterprise SP0.|||

Hi,

Apparently we have hit a bug where the usercache size becomes too large and for every execution of a statement SQL Server has to check this store.
This store is not cleaned and keep growing too the point where lookups become blocked with spinlocks. If you want I can give you the commands to check this.

This will be fixed in SP2 because of the major impact a fix would have. The workaround is adding the users to the sysadmin group or clearing the cache manually with DBCC FREESYSTEMCACHE.

HTH

Kind regards,
Wesley

|||Thanks for the replies !

Wes, if you could give the command to check the spinlocks that would be great.

many thanks

Stu|||

Stu,

It's actually the query for checking the cache size.

SELECT * FROM sys.dm_os_memory_clerks
WHERE type = 'USERSTORE_TOKENPERM'
We have seen problems as of 150MB in our environment but this probably all depends on the load and types of queries. Add up the single pages and multipages to determine the size.

HTH
Wesley

|||Thanks Wesley!

After much investigation it turned out the problem was the iLO Management Channel Interface Driver (Cpqcidrv.sys) from Hewlett-Packard. Apparently this

is known to cause this issue on x64 editions of SQL Server 2005. The driver

causes the SQL Server 2005 64-bit working set to be trimmed!


We do not use ILO on our HP Servers, so I disable it in device manager, and so far so good (fingers crossed)!!!

Thanks

Stu|||

Great news!

There seems to be a hotfix for Windows 2003 concerning working set trims too.
Apparently working sets get trimmed when you use Terminal Services on the server... so beware ;-)

|||

If possible could you share the hotfix number? Thanks in advance,

Drew

|||

Hey Wesley. Can you please elaborate more on this 'working sets get trimmed' and beware. and the hotfix available? Thank you very much. I've had problems with this also.

|||

I am also have a problem with this issue. I am going to try disabling the ILO. The terminal service hotfix seems to be this one:

http://support.microsoft.com/kb/905865/en-us

sqlsql

Cluster failing connections with SSL error message

We recently migrated our production environment from a win2k3/SQL 2000 EE cluster to a new 64 bit win2k3/ SQL 2005 SP1 Cluster.

Cluster works fine for a while (3-6 hours), then our logins start failing. We recieve the following error in the event log:

"The server was unable to load the SSL provider library needed to log in; the connection has been closed. SSL is used to encrypt either the login sequence or all communications, depending on how the administrator has configured the server. See Books Online for information on this error message"

The only way to resume connectivity is to offline and online the sql service in the cluster.

The only reference to this error that I can find has something to do with a hotfix that might be available?

It's sad to say but we are experiencing the same issue.
Only we get this after a good 30 minutes...

Someone from Microsoft Spain is flying over here.

Very painful

|||

In working with microsoft PSS, we found the issue to be a permissions issue of all things.

Our service account did not have rights to lock pages in memory on either machines in the cluster. (even though is a domain administrator).

Once we gave it this right and restarted the SQL Server object in the cluster we have not had a similar failure since.

|||

This was unfortunately enabled already so it did not cause the problem here :-(

|||Hi Wesley,

Any joy with this? I'm currently going throught the same thing, and also have locked pages enabled for my SQL service account. I'm currently going through MS Support, but seem to be going around the houses :o(

Thanks

Stu|||Any luck, we have a similar problem.
We noticed that memory usage grows slowly beyond physical memory (32G) then connection failiurs start and server becomes inoperable.
Workaround solution is to reduce max mem in the server, down to 18G from (25).
This causes the server and OS to release memory. After the system stabilized we increased the maxmem in the server back to its original settng. Lock pages in memory is set.

Hope this helps.

This is a workaround, so we don't have to bounce the server, we are still looking for the cause.
esk.

Setup:2 node Cluster 4*dual 64bit AMD, 32Gb, SQL 2005 Enterprise SP0.|||

Hi,

Apparently we have hit a bug where the usercache size becomes too large and for every execution of a statement SQL Server has to check this store.
This store is not cleaned and keep growing too the point where lookups become blocked with spinlocks. If you want I can give you the commands to check this.

This will be fixed in SP2 because of the major impact a fix would have. The workaround is adding the users to the sysadmin group or clearing the cache manually with DBCC FREESYSTEMCACHE.

HTH

Kind regards,
Wesley

|||Thanks for the replies !

Wes, if you could give the command to check the spinlocks that would be great.

many thanks

Stu|||

Stu,

It's actually the query for checking the cache size.

SELECT * FROM sys.dm_os_memory_clerks
WHERE type = 'USERSTORE_TOKENPERM'
We have seen problems as of 150MB in our environment but this probably all depends on the load and types of queries. Add up the single pages and multipages to determine the size.

HTH
Wesley

|||Thanks Wesley!

After much investigation it turned out the problem was the iLO Management Channel Interface Driver (Cpqcidrv.sys) from Hewlett-Packard. Apparently this

is known to cause this issue on x64 editions of SQL Server 2005. The driver

causes the SQL Server 2005 64-bit working set to be trimmed!


We do not use ILO on our HP Servers, so I disable it in device manager, and so far so good (fingers crossed)!!!

Thanks

Stu|||

Great news!

There seems to be a hotfix for Windows 2003 concerning working set trims too.
Apparently working sets get trimmed when you use Terminal Services on the server... so beware ;-)

|||

If possible could you share the hotfix number? Thanks in advance,

Drew

|||

Hey Wesley. Can you please elaborate more on this 'working sets get trimmed' and beware. and the hotfix available? Thank you very much. I've had problems with this also.

|||

I am also have a problem with this issue. I am going to try disabling the ILO. The terminal service hotfix seems to be this one:

http://support.microsoft.com/kb/905865/en-us

Cluster failing connections with SSL error message

We recently migrated our production environment from a win2k3/SQL 2000 EE cluster to a new 64 bit win2k3/ SQL 2005 SP1 Cluster.

Cluster works fine for a while (3-6 hours), then our logins start failing. We recieve the following error in the event log:

"The server was unable to load the SSL provider library needed to log in; the connection has been closed. SSL is used to encrypt either the login sequence or all communications, depending on how the administrator has configured the server. See Books Online for information on this error message"

The only way to resume connectivity is to offline and online the sql service in the cluster.

The only reference to this error that I can find has something to do with a hotfix that might be available?

It's sad to say but we are experiencing the same issue.
Only we get this after a good 30 minutes...

Someone from Microsoft Spain is flying over here.

Very painful

|||

In working with microsoft PSS, we found the issue to be a permissions issue of all things.

Our service account did not have rights to lock pages in memory on either machines in the cluster. (even though is a domain administrator).

Once we gave it this right and restarted the SQL Server object in the cluster we have not had a similar failure since.

|||

This was unfortunately enabled already so it did not cause the problem here :-(

|||Hi Wesley,

Any joy with this? I'm currently going throught the same thing, and also have locked pages enabled for my SQL service account. I'm currently going through MS Support, but seem to be going around the houses :o(

Thanks

Stu|||Any luck, we have a similar problem.
We noticed that memory usage grows slowly beyond physical memory (32G) then connection failiurs start and server becomes inoperable.
Workaround solution is to reduce max mem in the server, down to 18G from (25).
This causes the server and OS to release memory. After the system stabilized we increased the maxmem in the server back to its original settng. Lock pages in memory is set.

Hope this helps.

This is a workaround, so we don't have to bounce the server, we are still looking for the cause.
esk.

Setup:2 node Cluster 4*dual 64bit AMD, 32Gb, SQL 2005 Enterprise SP0.|||

Hi,

Apparently we have hit a bug where the usercache size becomes too large and for every execution of a statement SQL Server has to check this store.
This store is not cleaned and keep growing too the point where lookups become blocked with spinlocks. If you want I can give you the commands to check this.

This will be fixed in SP2 because of the major impact a fix would have. The workaround is adding the users to the sysadmin group or clearing the cache manually with DBCC FREESYSTEMCACHE.

HTH

Kind regards,
Wesley

|||Thanks for the replies !

Wes, if you could give the command to check the spinlocks that would be great.

many thanks

Stu|||

Stu,

It's actually the query for checking the cache size.

SELECT * FROM sys.dm_os_memory_clerks
WHERE type = 'USERSTORE_TOKENPERM'
We have seen problems as of 150MB in our environment but this probably all depends on the load and types of queries. Add up the single pages and multipages to determine the size.

HTH
Wesley

|||Thanks Wesley!

After much investigation it turned out the problem was the iLO Management Channel Interface Driver (Cpqcidrv.sys) from Hewlett-Packard. Apparently this

is known to cause this issue on x64 editions of SQL Server 2005. The driver

causes the SQL Server 2005 64-bit working set to be trimmed!


We do not use ILO on our HP Servers, so I disable it in device manager, and so far so good (fingers crossed)!!!

Thanks

Stu|||

Great news!

There seems to be a hotfix for Windows 2003 concerning working set trims too.
Apparently working sets get trimmed when you use Terminal Services on the server... so beware ;-)

|||

If possible could you share the hotfix number? Thanks in advance,

Drew

|||

Hey Wesley. Can you please elaborate more on this 'working sets get trimmed' and beware. and the hotfix available? Thank you very much. I've had problems with this also.

|||

I am also have a problem with this issue. I am going to try disabling the ILO. The terminal service hotfix seems to be this one:

http://support.microsoft.com/kb/905865/en-us

Cluster failing connections with SSL error message

We recently migrated our production environment from a win2k3/SQL 2000 EE cluster to a new 64 bit win2k3/ SQL 2005 SP1 Cluster.

Cluster works fine for a while (3-6 hours), then our logins start failing. We recieve the following error in the event log:

"The server was unable to load the SSL provider library needed to log in; the connection has been closed. SSL is used to encrypt either the login sequence or all communications, depending on how the administrator has configured the server. See Books Online for information on this error message"

The only way to resume connectivity is to offline and online the sql service in the cluster.

The only reference to this error that I can find has something to do with a hotfix that might be available?

It's sad to say but we are experiencing the same issue.
Only we get this after a good 30 minutes...

Someone from Microsoft Spain is flying over here.

Very painful

|||

In working with microsoft PSS, we found the issue to be a permissions issue of all things.

Our service account did not have rights to lock pages in memory on either machines in the cluster. (even though is a domain administrator).

Once we gave it this right and restarted the SQL Server object in the cluster we have not had a similar failure since.

|||

This was unfortunately enabled already so it did not cause the problem here :-(

|||Hi Wesley,

Any joy with this? I'm currently going throught the same thing, and also have locked pages enabled for my SQL service account. I'm currently going through MS Support, but seem to be going around the houses :o(

Thanks

Stu|||Any luck, we have a similar problem.
We noticed that memory usage grows slowly beyond physical memory (32G) then connection failiurs start and server becomes inoperable.
Workaround solution is to reduce max mem in the server, down to 18G from (25).
This causes the server and OS to release memory. After the system stabilized we increased the maxmem in the server back to its original settng. Lock pages in memory is set.

Hope this helps.

This is a workaround, so we don't have to bounce the server, we are still looking for the cause.
esk.

Setup:2 node Cluster 4*dual 64bit AMD, 32Gb, SQL 2005 Enterprise SP0.|||

Hi,

Apparently we have hit a bug where the usercache size becomes too large and for every execution of a statement SQL Server has to check this store.
This store is not cleaned and keep growing too the point where lookups become blocked with spinlocks. If you want I can give you the commands to check this.

This will be fixed in SP2 because of the major impact a fix would have. The workaround is adding the users to the sysadmin group or clearing the cache manually with DBCC FREESYSTEMCACHE.

HTH

Kind regards,
Wesley

|||Thanks for the replies !

Wes, if you could give the command to check the spinlocks that would be great.

many thanks

Stu|||

Stu,

It's actually the query for checking the cache size.

SELECT * FROM sys.dm_os_memory_clerks
WHERE type = 'USERSTORE_TOKENPERM'
We have seen problems as of 150MB in our environment but this probably all depends on the load and types of queries. Add up the single pages and multipages to determine the size.

HTH
Wesley

|||Thanks Wesley!

After much investigation it turned out the problem was the iLO Management Channel Interface Driver (Cpqcidrv.sys) from Hewlett-Packard. Apparently this

is known to cause this issue on x64 editions of SQL Server 2005. The driver

causes the SQL Server 2005 64-bit working set to be trimmed!


We do not use ILO on our HP Servers, so I disable it in device manager, and so far so good (fingers crossed)!!!

Thanks

Stu|||

Great news!

There seems to be a hotfix for Windows 2003 concerning working set trims too.
Apparently working sets get trimmed when you use Terminal Services on the server... so beware ;-)

|||

If possible could you share the hotfix number? Thanks in advance,

Drew

|||

Hey Wesley. Can you please elaborate more on this 'working sets get trimmed' and beware. and the hotfix available? Thank you very much. I've had problems with this also.

|||

I am also have a problem with this issue. I am going to try disabling the ILO. The terminal service hotfix seems to be this one:

http://support.microsoft.com/kb/905865/en-us

CLONING PACKAGES

I loved the DTS feature that allowed saving DTS package as a VB model. With a bit of coding you could generate a number of packages from one template.

Are there any analogous solutions for SSIS? I cannot find anything.

My goal is simple. I have an ETL step that transfers data from staging dimension table to the corresponding star schema table in the subject matter database. I have two types of packages for SCD type 1 and type 2. Do you have any suggestions on how I can clone packages so that I don’t have to go manually through each of them to replace certain items such as stored procedure names, etc.

Thank you!

Besides copying the .dtsx file and editing the copy to suit your needs?|||You can create a program to generate SSIS packages. If you want to reverse your current packages into C# code, take a look at http://www.ivolva.com/ssis_code_generator.html.|||You could also create a template for each type of load process and store them in the \Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems folder. I think that's the correct place.|||Phil Brammer: Yes, I want to be able to change all hard-coded string dynamically. Assuming I use standard naming convention all object names will have the same structure. All I want to do is loop through table names and replace appropriate strings within the existing template package to generate new packages on the fly.|||jwelch, that's an interesting solution. I'll try to play with you and let you know if it works. But it seems it's what I need.|||

LoveDanger wrote:

Phil Brammer: Yes, I want to be able to change all hard-coded string dynamically. Assuming I use standard naming convention all object names will have the same structure. All I want to do is loop through table names and replace appropriate strings within the existing template package to generate new packages on the fly.

Using expressions, you can build just ONE package to do this though.

Provided the structures of all of the tables (both sources and destinations) are all the same... That is, you can dynamically adjust the tables the OLE DB sources and destinations go against.|||Thanks Phil, this should work. I assume the transformation part would be tricky (I'm not sure how to map all the columns dynamically), but I'll check if that could be done. Also, I believe the solution posted by jwelch (http://www.ivolva.com/ssis_code_generator.html) should work if one still wants to generate numerous packages.

Thank you all!
|||Note: My solution works if you are looping through tables that have the same structure. If they don't, then my solution won't work. Once you build the package (mappings and all) you can start using expressions to dynamically change the table names.|||Yeah, in my case we have different set of attributes for each dimension that I'm not sure how to handle. I used your approach with BCP though - use only one package to transfer data from a number of flat files to SQL Server tables. Though you still have to generate format files for each that pretty much translates your column mappings.|||

LoveDanger wrote:

I loved the DTS feature that allowed saving DTS package as a VB model. With a bit of coding you could generate a number of packages from one template.

Are there any analogous solutions for SSIS? I cannot find anything.

My goal is simple. I have an ETL step that transfers data from staging dimension table to the corresponding star schema table in the subject matter database. I have two types of packages for SCD type 1 and type 2. Do you have any suggestions on how I can clone packages so that I don’t have to go manually through each of them to replace certain items such as stored procedure names, etc.

Thank you!

Sounds to me like you want to use templates. Matt explained where to drop them elsewhere in this thread.

-Jamie

|||

Well, I am not really sure how would templates solve my problem. I will still have to go through each package task and replace the names of the objects manually.

Does template allow you to do search and replace for the entire package?

|||

LoveDanger wrote:

Well, I am not really sure how would templates solve my problem. I will still have to go through each package task and replace the names of the objects manually.

It sounds as though you'll have to do that regardless of the solution though?

LoveDanger wrote:

Does template allow you to do search and replace for the entire package?

No, not really. You could open up the package's XML (right-click on the package and select 'View Code') and do a Find-Replace that way of that's what you want to do.

-Jamie

|||Thanks Jamie, I think this would be the easiest in my case. At least it will save time on going through each task manually.

Anastasia|||

LoveDanger wrote:

Thanks Jamie, I think this would be the easiest in my case. At least it will save time on going through each task manually.

Anastasia

Cool. Take a copy of the package before you alter it