Jump to content
McObject Forums
Sign in to follow this  
wilson0x4d

Incompatible key type (bug?)

Recommended Posts

wilson0x4d    0

We have a data file that was returning a strange error after being re-opened (failure to cast from ObjectA to ObjectB after fetched from a PArray, where ObjectA wasn't the type being stored to the PArray to begin with but was instead a child of ObjectB).

Before posting to the forums I wanted to have a console application that reproduced the error.

While I cannot repro the error, I am experiencing a different error every time I execute the test app. I am hoping that this error is somehow related to the other error, but even if it is not it's something I would like to get resolved and understand before moving forward.

Exception Details (verbatim):

Perst.StorageError occurred
_HResult=-2146233088
_message=Incompatible key type
HResult=-2146233088
IsTransient=false
Message=Incompatible key type
Source=PerstNetGenerics
StackTrace:
at Perst.Impl.Btree`2.checkKey(Key key) in d:\Perst4.NET\src\impl\Btree.cs:line 444
InnerException:
incidentally, there is a second exception we have experienced while running this tool:
System.NullReferenceException was unhandled
_HResult=-2147467261
_message=Object reference not set to an instance of an object.
HResult=-2147467261
IsTransient=false
Message=Object reference not set to an instance of an object.
Source=PerstNetGenerics
StackTrace:
at Perst.Impl.BtreeFieldIndex`2.extractKey(Object obj) in d:\Perst4.NET\src\impl\BtreeFieldIndex.cs:line 145
at Perst.Impl.BtreeFieldIndex`2.Put(V obj) in d:\Perst4.NET\src\impl\BtreeFieldIndex.cs:line 230
at ConsoleApplication1.TestContainer`1.Add(Guid key, TValue value) in z:\Code\wilson0x4d\Perst\ConsoleApplication1\ConsoleApplication1\TestContainer.cs:line 244
at ConsoleApplication1.Program.PopulatorThreadMain(Object state) in z:\Code\wilson0x4d\Perst\ConsoleApplication1\ConsoleApplication1\Program.cs:line 117
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
InnerException:
I'm not sure how to post the source code for testing this issue, it's consistent, though.

Share this post


Link to post
Share on other sites
wilson0x4d    0

I think I need help understanding something about how Perst functions internally.

I use reader/writer locks to protect access to individual FieldIndex<> instances, my belief was that this would provide less contention between individual indexes. Thus, there is one r/w lock per index.

However, I've introduced a BeginThreadTransaction(ReadWrite) and BeginThreadTransaction(ReadOnly) wherever I have a rwlock.EnterWriteLock and rwlock.EnterReadLock (respectively.)

This seems to have eliminated the error I was experiencing before, thus raising a question:

Despite having multiple FieldIndex<> instances are they still sharing some internal state and overwriting one another? (Such that I cannot hierarchically lock per each FieldIndex, but must resort to a global/shared lock which encompasses all indexes.)

If this is the case, is there anything I can do to isolate these indexes short of instancing multiple separate storage objects? I tried setting the perst.alternative.btree option but this didn't seem to resolve my issue. I also tried the multiclient option but this incurred the requirement that i use serializable transactions which resulted in much slower performance (by a factor of almost 200.)

One of the reasons this structure uses multiple FieldIndex<> instances is that we noticed index performance degredation somewhere between 300k and 500k inserts, and we have a requirement to store roughly half a billion items (~100 million a day.) We tested the persistentHash implementation and despite being faster it seemed to leave garbage in the store (we have a unit test which fills the data structure, removes the items, then performs a backup to compact the file.. and it does this 100 times. Unlike other data structures the persistent hash would grow with every iteration.)

At any rate, thanks for any additional info/confirmations/etc. :)

Share this post


Link to post
Share on other sites
wilson0x4d    0

Well, I may have spoke too soon.

Seems that in all cases I receive "incompatible key" error, in some configurations it's just "chance" that it occurs. In other configurations I can coerce it every time I run the test tool.

I have uploaded a new version of the test code that has three #define's at the top of the TestContainer.cs file, this makes it much easier to test the following:

1. .NET ReaderWriterLock for synchronization (#define USE_RWLOCK)

2. Storage.BeginThreadTransaction for synchronization (#define USE_PTRAN)

3. perst.alternative.btree property setting (#define USE_ALTERNATIVE_BTREE)

This code removes the "r/w lock per-index" which consistently failed with a null-reference exception. However, if someone tells me that it should work I would prefer such a locking model since it would dramatically reduce contention in our application.

New source code is: http://1drv.ms/1o98ByP

Again, thanks for any insight/help on this.

Share this post


Link to post
Share on other sites
perstmco    0

_buckets (PArray) is also Perst object and access to it should be also synchronized.
So you should acquire lock before accessing _bundle:

public void Add(Guid key, TValue value)
{

var bucket = key.Mod(_mod);
var _index = _buckets[bucket];

#if USE_RWLOCK
_rwlock.EnterWriteLock();
#endif
#if USE_PTRAN
Storage.BeginThreadTransaction(TransactionMode.ReadWrite);
#endif


===>

public void Add(Guid key, TValue value)
{
#if USE_RWLOCK
_rwlock.EnterWriteLock();
#endif
#if USE_PTRAN
Storage.BeginThreadTransaction(TransactionMode.ReadWrite);
#endif

var bucket = key.Mod(_mod);
var _index = _buckets[bucket];




Patched version of your example is attached

ConsoleApplication1.zip

Share this post


Link to post
Share on other sites
wilson0x4d    0

Thanks, I overlooked that since we only assign a value to PArray once.. but I can see how that doesn't mean the value isn't ever paged out and back in again (thus necessitating a lock.) We've integrated this change in our code.

We're still seeing this "Incompatible key type" error in our live environment. It happens once for every few thousand Get() calls.

I can also get the test code to exhibit this error "inconsistently", for example on the current run I successfully added ~138,000 entries before it occurred. I had to change the number of items being inserted in Program.cs (lines 16-17), then stopped and restarted the test a few times. Other times the app was able to insert over a million items without a problem. Inconsistent results.

The net effect for us is that when this error occurs we believe we are orphaning an object in the store, and/or failing to add an object to the store (depending on the exact call path from our code, it occurs in the case of TestContainer::Remove and TestContainer::Add).

From BTree.cs, Lines 442-445

                if (key.type != type)
                {
                    throw new StorageError(StorageError.ErrorCode.INCOMPATIBLE_KEY_TYPE);
                }

When this happens under the debugger I inspect "key.type" and "type" and they are both "tpGuid", I'm then confused why this exception is being thrown. Then I drag the "instruction pointer" back up to the conditional and re-execute and it succeeds (does not eval false the second time.) I looked at the code for getKeyFromObject and see that 'key' is (should be) a new object instance which (IMO) rules out there being a race condition on setting the 'type' property, and it also looks like BTree::type is set only once during construction as well?

I'm totally confused about this error.

Additionally, the null reference exception thrown by BTreeFieldIndex is also exhibiting similar behavior:

From BTreeFieldIndex.cs Line 145:

        private Key extractKey(object obj) 
        { 
            Object val = mbr is FieldInfo ? ((FieldInfo)mbr).GetValue(obj) : ((PropertyInfo)mbr).GetValue(obj, null);
            if (val == null)

When I eval "mbr is FieldInfo ? ((FieldInfo)mbr).GetValue(obj) : ((PropertyInfo)mbr).GetValue(obj, null)" in the watch window it properly extracts the key value.

If I continue the debug session everything seems to go on as if nothing happened.

I can work around these issues by updating our calling code to retry until it works, but this seems like a horribly inefficient hack. We would much rather understand root cause and fix it or stop doing whatever it is we're doing that is causing it (assuming it's our code/usage of the code.)

That this is reproducable on 3 different machines (laptop, desktop, server hardware) and from two different EXEs (service EXE and this Console EXE) I have to rule out hardware and software environment being the cause. We've confirmed the issue on Windows 2012, Windows 8 and Windows 8.1 though I doubt these things matter. Additionally, all our test hardware has a minimum of 4 cores, but again don't know that this matters.

Thanks again for any insight/assistance.

Share this post


Link to post
Share on other sites
perstmco    0

>> When this happens under the debugger I inspect "key.type" and "type" and they are both "tpGuid", I'm then confused why this exception is being thrown.

This is because of a race condition. Two or more threads try to concurrently access the index and, as a result, get this exception. It means that you have not protected all accesses to _bucket by locks. Can you send the updated version of your example so that we can verify it and try to reproduce the problem?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×