Combine lemon zest, lemon juice, shallot, tomatoes, oil, vinegar, salt, black pepper, Worcestershire sauce, cayenne, and 1 Tbsp. horseradish in a large bowl. Let marinate 10 minutes, then stir in celery stalks and leaves.
Meanwhile, combine mayonnaise and remaining 4 Tbsp. horseradish in a small bowl. Spread 1 Tbsp. horseradish-mayonnaise on each slice of rye. Top toasts evenly with tomato mixture.