[PSA] Unraveling "bll cannot be cast to fv" / invalid chunk offsets in region files

So, recently, on a server I play on, bad things have started happening. Players will reach certain coordinates and get kicked for an internal server error as well as each time they try to log in again. We see their join and leave messages in game. The way to fix it is changing their coordinates.

Then I got in contact with the server dev and he sent me a stack trace for the error shown in console:

Code:

CONSOLE ERROR:
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.ThreadServerApplication.run(SourceFile:628) [craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.MinecraftServer.run(MinecraftServer.java:469) [craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.MinecraftServer.u(MinecraftServer.java:558) [craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.DedicatedServer.v(DedicatedServer.java:260) [craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.MinecraftServer.v(MinecraftServer.java:667) [craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.ServerConnection.c(SourceFile:134) [craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.NetworkManager.a(NetworkManager.java:157) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PacketPlayInPosition.handle(SourceFile:63) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PacketPlayInFlying.a(SourceFile:137) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PlayerConnection.a(PlayerConnection.java:417) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PlayerList.d(PlayerList.java:297) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PlayerChunkMap.movePlayer(PlayerChunkMap.java:243) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PlayerChunkMap.b(PlayerChunkMap.java:167) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PlayerChunkMap.a(PlayerChunkMap.java:91) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.PlayerChunk.<init>(PlayerChunk.java:35) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:111) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.ChunkRegionLoader.chunkExists(ChunkRegionLoader.java:42) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.RegionFileCache.a(SourceFile:60) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at net.minecraft.server.v1_7_R3.RegionFile.<init>(RegionFile.java:76) ~[craftbukkit3092.jar:git-Bukkit-1.7.9-R0.1-10-g8688bd4-b3092jnks]
16.07 07:50:01 [Server] INFO at java.util.ArrayList.set(Unknown Source) ~[?:1.7.0_51]
16.07 07:50:01 [Server] INFO at java.util.ArrayList.elementData(Unknown Source) ~[?:1.7.0_51]
16.07 07:50:01 [Server] INFO java.lang.ArrayIndexOutOfBoundsException: -3901051
16.07 07:50:01 [Server] INFO [07:50:01 WARN]: Failed to handle packet for /ip-address-removed-here

It's throwing an ArrayIndexOutOfBoundsException here:
https://github.com/Bukkit/CraftBukk...java/net/minecraft/server/RegionFile.java#L76

Doing some research, the value "-3901051" is supposed to be the offset in the region file where the chunk is located. It would later be multiplied by 4096 to get the offset in bytes. Yikes.

Apparently this sort of corruption of the chunk offset table has happened before on the server and the map had to be run through a world repair tool. Unfortunately the problem has not gone away despite keeping plugins and CB up to date.

Because the exception occurs when the region is loaded, that means that an entire region is corrupt from this one invalid offset. My best guess as to how it got corrupt in the first place is a plugin calling the wrong obfuscated method.

It's completely impractical to try removing individual plugins to see if the problem goes away - it happens over time and at random and on occasion seems to even fix itself temporarily. Googling the error from the kick screen, it seems that this is quite a commonplace issue for which the solution is not known.



So, the only thing I can validly ask is, does anyone know of likely candidates for the cause of the corruption? There is little more info I can provide and little I can do since I'm just a normal member of the server. Any information about what the cause could be would be helpful - I don't want to see this world get corrupted region by region.

Alternatively, are there any recommendations for modifying the linked code so as to pretend that chunk was not generated and continue loading the region? Losing a single chunk is better than losing an entire region.

For now, the server dev has opted to take the server offline for quite some time to run the worlds through a world repair tool that was used the last time the issue surfaced.

 

My recommendation for this point alone would be to actually check the source of the plugins to see which one is doing it.

 

Would do, but I'm not privy to the information of exactly what plugins are being used and what versions. Thanks for the idea though, if I ever do become privy to that information I'll try that suggestion.